Oct 31 00:53:43.918041 ip-10-0-136-249 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 00:53:43.918050 ip-10-0-136-249 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 00:53:43.918058 ip-10-0-136-249 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 00:53:43.918280 ip-10-0-136-249 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 00:53:54.069868 ip-10-0-136-249 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 00:53:54.069882 ip-10-0-136-249 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 33d6cd26724d4abb9bd23cc8c8bdc862 -- Oct 31 00:55:53.337172 ip-10-0-136-249 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:55:53.736736 ip-10-0-136-249 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:53.736736 ip-10-0-136-249 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:55:53.736736 ip-10-0-136-249 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:53.736736 ip-10-0-136-249 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:55:53.741930 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741808 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.741930 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741930 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.741930 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741934 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741938 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741941 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741945 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741947 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741950 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741953 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741955 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741958 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741961 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741963 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741966 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741968 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741971 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741974 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741976 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741979 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741981 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741984 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.741986 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.742030 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742001 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742004 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742007 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742010 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742012 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742015 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742017 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742020 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742022 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742025 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742028 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742030 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742033 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742035 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742039 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742042 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742044 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742047 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742049 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742052 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.742519 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742055 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742057 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742060 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742063 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742065 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742069 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742073 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742076 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742079 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742082 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742085 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742087 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742090 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742094 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742098 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742100 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742103 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742106 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742109 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.742999 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742111 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742114 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742116 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742119 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742121 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742124 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742126 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742129 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742131 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742134 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742136 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742138 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742141 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742144 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742146 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742149 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742151 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742154 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742157 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742159 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.743461 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742162 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742164 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742167 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742169 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742172 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742175 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742180 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742185 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742188 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742191 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742194 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742196 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742199 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742201 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742204 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742206 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742208 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742211 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742213 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.744011 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742216 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742218 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742220 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742223 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742225 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.742228 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743014 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743021 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743024 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743026 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743029 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743032 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743035 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743038 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743040 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743043 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743045 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743048 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743051 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743054 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.744452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743056 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743060 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743062 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743065 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743068 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743070 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743073 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743075 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743078 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743080 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743082 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743085 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743087 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743090 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743092 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743095 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743097 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743099 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743102 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743105 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.744937 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743110 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743112 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743115 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743118 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743121 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743124 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743126 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743130 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743134 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743137 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743140 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743142 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743145 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743148 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743150 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743152 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743155 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743157 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.745412 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743160 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743162 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743165 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743167 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743170 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743172 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743177 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743180 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743183 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743185 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743188 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743190 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743193 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743196 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743198 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743202 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743204 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743207 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743209 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.745855 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743212 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743215 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743217 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743220 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743222 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743225 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743227 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743229 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743232 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743235 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743237 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743240 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743242 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743245 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743247 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743250 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743252 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743255 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743257 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743260 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.746356 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743262 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743265 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743267 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743269 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743272 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743274 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743276 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743279 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743287 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743289 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743295 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743297 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743300 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743815 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743820 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743824 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743826 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743829 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743831 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.746829 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743834 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743837 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743840 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743842 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743845 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743847 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743850 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743853 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743855 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743872 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743875 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743878 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743880 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743883 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743885 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743888 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743890 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743893 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743895 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.747318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743898 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743903 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743906 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743908 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743911 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743914 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743916 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743919 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743921 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743923 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743926 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743928 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743931 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743934 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743936 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743939 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743942 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743945 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743947 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.747769 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743950 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743952 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743955 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743957 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743959 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743962 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743964 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743967 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743969 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743972 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743974 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743977 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743980 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743982 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743986 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743989 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743992 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743994 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743997 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.743999 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.748227 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744002 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744004 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744007 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744009 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744012 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744015 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744017 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744020 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744023 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744025 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744028 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744031 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744033 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744036 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744039 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744041 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744043 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744046 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744050 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.748697 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744053 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744056 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744059 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744062 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744064 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744067 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744069 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744073 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744078 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744082 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744085 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744088 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744090 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744092 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744095 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744097 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744100 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744102 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744105 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.749153 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744109 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744112 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.744114 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745075 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745085 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745092 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745098 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745102 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745106 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745110 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745115 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745118 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745122 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745125 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745128 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745131 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745134 2578 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745137 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745149 2578 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745153 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745156 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745160 2578 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:55:53.749601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745163 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745166 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745169 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745173 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745177 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745179 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745182 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745185 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745188 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745191 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745194 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745197 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745200 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745204 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745208 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745211 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745213 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745216 2578 flags.go:64] FLAG: --enable-server="true" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745219 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745223 2578 flags.go:64] FLAG: --event-burst="100" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745227 2578 flags.go:64] FLAG: --event-qps="50" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745230 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745233 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745236 2578 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:55:53.750217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745240 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745243 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745246 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745249 2578 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745252 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745255 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745258 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745261 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745264 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745267 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745270 2578 flags.go:64] FLAG: --feature-gates="" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745274 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745277 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745280 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745283 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745286 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745289 2578 flags.go:64] FLAG: --help="false" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745292 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745296 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745299 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745301 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745304 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745308 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:55:53.750796 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745312 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745315 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745317 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745320 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745323 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745326 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745329 2578 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745332 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745334 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745337 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745340 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745343 2578 flags.go:64] FLAG: --lock-file="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745346 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745349 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745352 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745357 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745360 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745363 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745366 2578 flags.go:64] FLAG: --logging-format="text" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745368 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745372 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745374 2578 flags.go:64] FLAG: --manifest-url="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745377 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745382 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745385 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:55:53.751350 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745388 2578 flags.go:64] FLAG: --max-pods="110" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745391 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745395 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745398 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745400 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745403 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745406 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745409 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745416 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745420 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745423 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745426 2578 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745431 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745434 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745437 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745440 2578 flags.go:64] FLAG: --port="10250" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745443 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745445 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-07c8891944f7ab7ce" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745448 2578 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745451 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745454 2578 flags.go:64] FLAG: --register-node="true" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745457 2578 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745461 2578 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745464 2578 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745466 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745469 2578 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:55:53.751990 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745473 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745476 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745478 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745481 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745484 2578 flags.go:64] FLAG: --runonce="false" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745487 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745490 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745492 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745495 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745498 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745501 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745504 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745507 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745509 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745512 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745515 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745518 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745521 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745525 2578 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745528 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745534 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745536 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745539 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745544 2578 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745546 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:55:53.752637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745551 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745554 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745557 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745560 2578 flags.go:64] FLAG: --v="2" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745565 2578 flags.go:64] FLAG: --version="false" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745569 2578 flags.go:64] FLAG: --vmodule="" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745572 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.745576 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745712 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745716 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745719 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745721 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745724 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745726 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745728 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745731 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745733 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745736 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745739 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745741 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745743 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.753230 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745746 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745748 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745751 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745754 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745757 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745760 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745763 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745765 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745768 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745770 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745772 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745776 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745779 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745781 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745784 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745786 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745788 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745791 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745794 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.753717 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745798 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745801 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745804 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745806 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745809 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745812 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745814 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745817 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745819 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745822 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745824 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745827 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745829 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745831 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745834 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745837 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745840 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745843 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745846 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745848 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.754192 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745851 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745853 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745855 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745872 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745876 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745879 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745882 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745885 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745887 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745890 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745893 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745895 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745898 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745900 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745903 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745905 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745908 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745910 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745913 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.754646 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745915 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745918 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745921 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745923 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745925 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745928 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745931 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745933 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745936 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745938 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745942 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745945 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745947 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745950 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745952 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745955 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745958 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745962 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745965 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.755147 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745967 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745970 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745972 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745975 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745977 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745980 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745982 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745985 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745988 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745990 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745993 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745995 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.745998 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746002 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746005 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746008 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.755594 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.746035 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746071 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746077 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746080 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746084 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746087 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746089 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746093 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746096 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746098 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746101 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746103 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746106 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746108 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746111 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746114 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746117 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746119 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746122 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746124 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.755992 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746127 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746129 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746131 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746134 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746139 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746142 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746144 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746147 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746149 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746151 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746154 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746157 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746159 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746162 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746165 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746170 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746173 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746176 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.756475 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746179 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746182 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746185 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746188 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746191 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746193 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746196 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746198 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746202 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746204 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746207 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746209 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746212 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746214 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746217 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746220 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746222 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746224 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746227 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.756933 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746229 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746232 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746234 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746237 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746240 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746242 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746245 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746247 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746249 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746252 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746254 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746257 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746260 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746263 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746266 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746268 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746271 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746274 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746277 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.757377 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746279 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746282 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746286 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746288 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746291 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746293 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746296 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746298 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746301 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746303 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746306 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746308 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746311 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746313 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746316 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746319 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746321 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746323 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746327 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746329 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.757833 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746332 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746334 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746337 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746339 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746344 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746350 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746353 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746355 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.746358 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.753460 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.753474 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.753489 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.753493 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753550 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753555 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753558 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753561 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753564 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753566 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.758318 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753569 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753572 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753574 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753577 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753580 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753582 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753585 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753588 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753591 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753594 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753596 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753599 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753602 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753604 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753607 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753609 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753612 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753614 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753617 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753619 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.758789 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753622 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753624 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753627 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753630 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753632 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753635 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753637 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753640 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753642 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753645 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753647 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753650 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753652 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753654 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753657 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753659 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753662 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753664 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753668 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.759346 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753672 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753675 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753677 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753680 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753683 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753685 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753687 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753690 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753692 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753695 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753697 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753699 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753702 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753704 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753707 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753710 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753713 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753715 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753718 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.759776 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753721 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753724 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753726 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753729 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753731 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753734 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753736 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753738 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753741 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753743 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753746 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753749 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753751 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753753 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753756 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753759 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753762 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753764 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753766 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753769 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.760271 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753771 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753774 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753776 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753779 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753781 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753783 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753786 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753788 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753791 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753794 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753796 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753799 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753801 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753804 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753806 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753808 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753811 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753813 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753817 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753820 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.760749 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753822 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.753825 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.753851 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754011 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754015 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754018 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754021 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754023 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754026 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754029 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754031 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754034 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754036 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754039 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754041 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754044 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.761242 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754046 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754049 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754051 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754053 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754056 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754059 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754061 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754064 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754066 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754069 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754071 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754073 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754076 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754078 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754080 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754083 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754085 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754087 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754090 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754092 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.761617 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754094 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754097 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754100 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754102 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754104 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754107 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754109 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754112 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754114 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754117 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754119 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754121 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754124 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754126 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754129 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754131 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754138 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754141 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.762332 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754144 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754147 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754149 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754152 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754154 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754157 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754159 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754161 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754164 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754166 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754168 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754171 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754173 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754175 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754178 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754180 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754183 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754185 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754187 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754190 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.763076 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754193 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754200 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754204 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754207 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754210 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754213 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754216 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754219 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754222 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754224 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754227 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754229 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754232 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754234 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754237 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754239 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754242 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754244 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.763543 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754246 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754249 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754251 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754253 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754256 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754258 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754260 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754263 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754265 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754267 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754270 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754272 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754274 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754277 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754280 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.754302 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:55:53.764050 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754332 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754338 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754341 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754344 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754346 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754349 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754352 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754354 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754357 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754360 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754362 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754365 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754367 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754369 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754372 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754374 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754377 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754379 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754382 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754384 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:55:53.764455 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754386 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754389 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754391 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754393 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754398 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754402 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754405 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754408 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754410 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754412 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754415 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754418 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754421 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754423 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754425 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754428 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754430 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754432 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754435 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:55:53.764918 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754437 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754439 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754442 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754444 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754446 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754449 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754451 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754453 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754456 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754458 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754461 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754463 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754465 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754468 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754470 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754472 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754475 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754477 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754480 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754482 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:55:53.765386 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754484 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754486 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754489 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754491 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754494 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754497 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754499 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754501 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754504 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754507 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754509 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754512 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754514 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754516 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754519 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754521 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754523 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754526 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754528 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:55:53.765830 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754530 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754533 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754535 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754537 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754540 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754542 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754545 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754547 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754549 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754551 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754554 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754556 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754559 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754562 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754564 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754569 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754571 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754574 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754577 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754579 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:55:53.766431 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754582 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754584 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754586 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754589 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754591 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754594 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754596 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:53.754599 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.754647 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.757567 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.758539 2578 server.go:1014] "Starting client certificate rotation" Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.759029 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.759650 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.760713 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:55:53.766900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.766142 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:55:53.777532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.777515 2578 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:55:53.781410 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.781393 2578 log.go:25] "Validated CRI v1 image API" Oct 31 00:55:53.782229 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.782201 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:55:53.785808 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.785787 2578 fs.go:135] Filesystem UUIDs: map[2eec57ee-b68f-4c32-b47f-34294ad357f9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f61103ec-55f2-4caa-bfa2-ec694fe8eb8a:/dev/nvme0n1p3] Oct 31 00:55:53.785880 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.785808 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 00:55:53.791111 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.791002 2578 manager.go:217] Machine: {Timestamp:2026-10-31 00:55:53.789553442 +0000 UTC m=+0.338360067 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121520 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229d2cd4af3340095fc9ae053616c3 SystemUUID:ec229d2c-d4af-3340-095f-c9ae053616c3 BootID:33d6cd26-724d-4abb-9bd2-3cc8c8bdc862 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:24:7f:2d:01:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:24:7f:2d:01:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:d1:5f:c4:69:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:55:53.791178 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.791113 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:55:53.791205 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.791189 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:55:53.794240 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.794213 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:55:53.794368 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.794243 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-249.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:55:53.794410 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.794371 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:55:53.794410 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.794379 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:55:53.794462 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.794413 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:55:53.794462 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.794431 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:55:53.795548 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.795538 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:55:53.799824 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.799812 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:55:53.799954 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.799945 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:55:53.800795 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.800787 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:55:53.800825 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.800800 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:55:53.800825 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.800811 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:55:53.800825 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.800821 2578 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:55:53.800923 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.800829 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:55:53.801224 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.801209 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:55:53.803601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.803583 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:55:53.810135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.810117 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:55:53.810970 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.810958 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:55:53.811011 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.810988 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:55:53.811160 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811150 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:55:53.811213 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811166 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:55:53.811213 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811172 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:55:53.811213 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811178 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:55:53.811213 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811183 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:55:53.811213 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811189 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:55:53.811892 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811884 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:55:53.811927 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811893 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:55:53.811927 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811900 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:55:53.811927 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811906 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:55:53.811927 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811914 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:55:53.811927 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.811922 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:55:53.812671 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.812663 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:55:53.812700 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.812672 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:55:53.816467 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.816455 2578 server.go:1263] "Started kubelet" Oct 31 00:55:53.816556 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.816525 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:55:53.816606 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.816562 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:55:53.816644 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.816631 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:55:53.817308 ip-10-0-136-249 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:55:53.818075 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.818056 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:55:53.818820 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.818805 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:55:53.822803 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.822785 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 00:55:53.825131 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.825111 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:55:53.825773 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.825757 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:55:53.825885 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.825851 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:53.826511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826492 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:55:53.826511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826512 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:55:53.826655 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826598 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:55:53.826708 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826657 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:55:53.826708 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826659 2578 factory.go:55] Registering systemd factory Oct 31 00:55:53.826708 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826664 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:55:53.826708 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826681 2578 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:55:53.826921 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826875 2578 factory.go:153] Registering CRI-O factory Oct 31 00:55:53.826921 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826885 2578 factory.go:223] Registration of the crio container factory successfully Oct 31 00:55:53.827018 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826922 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:55:53.827018 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826936 2578 factory.go:103] Registering Raw factory Oct 31 00:55:53.827018 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.826946 2578 manager.go:1199] Started watching for new ooms in manager Oct 31 00:55:53.827993 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.827977 2578 manager.go:319] Starting recovery of all containers Oct 31 00:55:53.838487 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.838472 2578 manager.go:324] Recovery completed Oct 31 00:55:53.843005 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.842983 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:53.844522 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.844504 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:53.844565 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.844544 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:53.844565 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.844561 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:53.845087 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845076 2578 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:55:53.845087 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845085 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:55:53.845208 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845101 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:55:53.845208 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845167 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:53.845810 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845796 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:53.845906 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845817 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:53.845906 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.845828 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:53.847216 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.847205 2578 policy_none.go:50] "Start" Oct 31 00:55:53.847263 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.847220 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:55:53.847263 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.847230 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:55:53.849238 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.849228 2578 policy_none.go:44] "Start" Oct 31 00:55:53.899597 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.899447 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:55:53.903757 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.903744 2578 manager.go:341] "Starting Device Plugin manager" Oct 31 00:55:53.903840 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.903771 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:55:53.903840 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.903780 2578 server.go:87] "Starting device plugin registration server" Oct 31 00:55:53.904003 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.903992 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:55:53.904048 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.904004 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:55:53.904135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.904120 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:55:53.904233 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.904216 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:55:53.904233 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.904229 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:55:53.904703 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.904683 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:55:53.904799 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.904741 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:53.957289 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.957268 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-249.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 00:55:53.959292 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.957264 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-249.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 00:55:53.976723 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.976687 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:55:53.977982 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.977965 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:55:53.978066 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.977990 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:55:53.978066 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:53.978015 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:55:53.978155 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:53.978053 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 00:55:54.004780 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.004728 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:54.005543 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.005526 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:54.005641 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.005562 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:54.005641 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.005577 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:54.005641 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.005609 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.006731 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.005810 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-249.us-west-1.compute.internal.18e37736c98470cd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-249.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-249.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:55:53.816428749 +0000 UTC m=+0.365235374,LastTimestamp:2026-10-31 00:55:53.816428749 +0000 UTC m=+0.365235374,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-249.us-west-1.compute.internal,}" Oct 31 00:55:54.037476 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.037452 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:55:54.066666 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.066639 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wrrgk" Oct 31 00:55:54.075061 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.075045 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wrrgk" Oct 31 00:55:54.079056 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.079033 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal"] Oct 31 00:55:54.079153 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.079098 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:54.080112 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.080095 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:54.080197 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.080119 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:54.080197 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.080129 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:54.082357 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.082345 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:54.082497 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.082483 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.082530 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.082514 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:54.083436 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.083421 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:54.083500 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.083435 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:54.083500 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.083449 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:54.083500 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.083455 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:54.083500 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.083459 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:54.083500 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.083465 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:54.085642 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.085628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.085703 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.085654 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:54.086481 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.086465 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:54.086529 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.086493 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:54.086529 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.086507 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:54.113454 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.113437 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.113966 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.113943 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-249.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 00:55:54.114054 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.113896 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-249.us-west-1.compute.internal.18e37736cb312dae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-249.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-249.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-249.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:55:53.84452651 +0000 UTC m=+0.393333146,LastTimestamp:2026-10-31 00:55:53.84452651 +0000 UTC m=+0.393333146,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-249.us-west-1.compute.internal,}" Oct 31 00:55:54.114054 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.113994 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 00:55:54.114307 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.114268 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 00:55:54.116240 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.116223 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.127164 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.127125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.127251 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.127169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.127349 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.127337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e548c58b36505791ab3955482325fe49-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"e548c58b36505791ab3955482325fe49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.228275 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.228247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.228406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.228278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.228406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.228297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e548c58b36505791ab3955482325fe49-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"e548c58b36505791ab3955482325fe49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.228406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.228326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e548c58b36505791ab3955482325fe49-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"e548c58b36505791ab3955482325fe49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.228406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.228340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.228406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.228355 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.269779 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.269714 2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-136-249.us-west-1.compute.internal\" not found" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.269779 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.269765 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:54.321806 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.321777 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-249.us-west-1.compute.internal\" not found" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.417137 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.417110 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.419233 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.419215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.470110 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.470085 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:54.471166 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.471142 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:54.471276 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.471178 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:54.471276 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.471193 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:54.471276 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.471226 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.528691 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.528630 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:54.528691 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.528654 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-249.us-west-1.compute.internal\": node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:54.644562 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.644531 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:54.745472 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.745430 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:54.758599 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.758578 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 00:55:54.758762 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:54.758739 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 00:55:54.758835 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.758730 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a747793b6d5cd47a4aa82ac69821d294-e9c03ac64371c6c0.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-249.us-west-1.compute.internal.18e37736cb319116\": read tcp 10.0.136.249:59854->3.146.167.38:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-249.us-west-1.compute.internal.18e37736cb319116 default 5133 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-249.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-249.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-249.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:55:53 +0000 UTC,LastTimestamp:2026-10-31 00:55:54.005568841 +0000 UTC m=+0.554375467,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-249.us-west-1.compute.internal,}" Oct 31 00:55:54.846473 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.846393 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:54.946807 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:54.946768 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.046983 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.046958 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.074975 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.074958 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:55:55.075885 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.075851 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:55:55.075968 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.075898 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:55:55.075968 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.075909 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:55:55.079560 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.079537 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:50:54 +0000 UTC" deadline="2028-06-04 09:07:46.154815723 +0000 UTC" Oct 31 00:55:55.079619 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.079561 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13976h11m51.075258337s" Oct 31 00:55:55.080226 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.080213 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.147823 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.147767 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.248252 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.248215 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.349002 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.348978 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.449525 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.449448 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.498927 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:55.498884 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b884c31d96cb5a9a71220f2bda573a2.slice/crio-858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d WatchSource:0}: Error finding container 858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d: Status 404 returned error can't find the container with id 858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d Oct 31 00:55:55.499128 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:55.499104 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode548c58b36505791ab3955482325fe49.slice/crio-dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4 WatchSource:0}: Error finding container dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4: Status 404 returned error can't find the container with id dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4 Oct 31 00:55:55.503443 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.503427 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:55:55.549930 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.549902 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:55:55.610278 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.610258 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.627322 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.627300 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:55.694417 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.694397 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:55.696325 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.696311 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:55:55.762965 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.762905 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:55:55.804034 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.803984 2578 apiserver.go:52] "Watching apiserver" Oct 31 00:55:55.826522 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.826492 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 00:55:55.872488 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.872456 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.872648 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.872503 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:55:55.927846 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.927819 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:55:55.928181 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.928158 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-48ddr","openshift-image-registry/node-ca-vrqjp","openshift-network-operator/iptables-alerter-nzlwl","openshift-ovn-kubernetes/ovnkube-node-kmmnx","openshift-dns/node-resolver-649pf","openshift-multus/multus-zdntg","openshift-network-diagnostics/network-check-target-blwrh","kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx","kube-system/konnectivity-agent-lttht","openshift-cluster-node-tuning-operator/tuned-gbk88","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-ztqdl","openshift-multus/network-metrics-daemon-pgqj5"] Oct 31 00:55:55.932208 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.932190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:55.932285 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.932260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:55:55.934319 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.934303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:55.936528 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.936505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:55.939465 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.939448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:55.943478 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.943461 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:55.945759 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.945741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zdntg" Oct 31 00:55:55.949066 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.949037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:55.951396 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.951379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:55.951486 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.951464 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:55:55.953733 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.953716 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:55.956219 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.956201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:55.958794 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.958779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:55.961369 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.961349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:55.961447 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:55.961424 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:55:55.982784 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.982738 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"e548c58b36505791ab3955482325fe49","Type":"ContainerStarted","Data":"dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4"} Oct 31 00:55:55.983992 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.983971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"7b884c31d96cb5a9a71220f2bda573a2","Type":"ContainerStarted","Data":"858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d"} Oct 31 00:55:55.992459 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.992437 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:55:55.992459 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.992439 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:55:55.992757 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.992746 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:55:55.992922 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.992909 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:55:55.993137 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.993123 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:55:55.993829 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.993812 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6sjwh\"" Oct 31 00:55:55.994045 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.994029 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:55.994045 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.994036 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:55:55.994181 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.994079 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lhvrj\"" Oct 31 00:55:55.994181 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.994093 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:55:55.999389 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:55.999372 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:55:56.000212 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000187 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fksdd\"" Oct 31 00:55:56.000411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000392 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:55:56.000586 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000570 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:55:56.000586 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000573 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:55:56.000736 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000722 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.000777 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000741 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.000822 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.000802 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.001651 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.001637 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.002741 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.002727 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g8n5b\"" Oct 31 00:55:56.003060 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.003046 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:55:56.003192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.003179 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-98n4j\"" Oct 31 00:55:56.006054 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.006034 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:55:56.006143 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.006079 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.006143 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.006130 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfh75\"" Oct 31 00:55:56.028547 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.028498 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:55:56.033477 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.033461 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ndfs4\"" Oct 31 00:55:56.033550 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.033523 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.033550 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.033526 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:55:56.033651 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.033525 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:55:56.034046 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.034031 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.036906 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.036890 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:55:56.039230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-hostroot\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.039276 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039237 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-kubelet-config\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.039276 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039256 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-config\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039276 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-conf\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.039425 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-lib-modules\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.039425 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63419a67-aaca-408d-bcfa-7c302b749c9d-host\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.039425 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-kubelet\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039513 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-var-lib-kubelet\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.039513 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.039513 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-slash\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039660 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-node-log\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039660 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-netd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039660 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-device-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.039660 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577tv\" (UniqueName: \"kubernetes.io/projected/e951e6b3-30a9-4181-912e-1a40f1725862-kube-api-access-577tv\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.039832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.039832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-systemd-units\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-systemd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.039832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039811 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:56.039832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-conf-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-multus-daemon-config\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10f64c92-95d2-4fc4-add2-d174ec7fe24f-agent-certs\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-env-overrides\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-tmp\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.039971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnx6j\" (UniqueName: \"kubernetes.io/projected/d25f0afe-e3c3-4ded-ab76-c98083607e36-kube-api-access-qnx6j\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.040043 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-dbus\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-system-cni-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-hosts-file\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-tmp-dir\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-modprobe-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysconfig\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-etc-kubernetes\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040194 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10f64c92-95d2-4fc4-add2-d174ec7fe24f-konnectivity-ca\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn28c\" (UniqueName: \"kubernetes.io/projected/4ff778a1-143c-471d-a387-3897770b9502-kube-api-access-vn28c\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-sys-fs\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040366 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhvt\" (UniqueName: \"kubernetes.io/projected/0fda59f9-289b-431d-ba24-c291169fe1ae-kube-api-access-gdhvt\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25f0afe-e3c3-4ded-ab76-c98083607e36-iptables-alerter-script\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.040403 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-bin\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-socket-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63419a67-aaca-408d-bcfa-7c302b749c9d-serviceca\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040478 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-log-socket\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-netns\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-netns\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-etc-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-ovn\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff778a1-143c-471d-a387-3897770b9502-ovn-node-metrics-cert\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-registration-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-host\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040734 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pphfn\" (UniqueName: \"kubernetes.io/projected/3d090b9a-abb1-48c3-9a43-415165ea4c04-kube-api-access-pphfn\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-cnibin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-socket-dir-parent\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.040958 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzsz8\" (UniqueName: \"kubernetes.io/projected/63419a67-aaca-408d-bcfa-7c302b749c9d-kube-api-access-nzsz8\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lts\" (UniqueName: \"kubernetes.io/projected/1f47fdf3-f591-4184-abff-d4670f404937-kube-api-access-w2lts\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-tuned\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-k8s-cni-cncf-io\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-bin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-var-lib-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-run\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.040977 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8cv\" (UniqueName: \"kubernetes.io/projected/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kube-api-access-gj8cv\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-cnibin\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-kubernetes\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041089 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-sys\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25f0afe-e3c3-4ded-ab76-c98083607e36-host-slash\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-cni-binary-copy\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-multus\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-kubelet\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.041580 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041317 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-system-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-multus-certs\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041347 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-script-lib\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.041874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-systemd\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.042341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.042004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d2k2\" (UniqueName: \"kubernetes.io/projected/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-kube-api-access-4d2k2\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.043246 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.043226 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.044169 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.044152 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gr6qq\"" Oct 31 00:55:56.044692 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.044675 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr8df\"" Oct 31 00:55:56.047211 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.047197 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:55:56.047282 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.047208 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:55:56.047324 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.047296 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 00:55:56.047381 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.047349 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2j5vr" Oct 31 00:55:56.142608 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-dbus\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-system-cni-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-hosts-file\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-tmp-dir\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142671 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-modprobe-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysconfig\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-etc-kubernetes\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10f64c92-95d2-4fc4-add2-d174ec7fe24f-konnectivity-ca\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.142787 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn28c\" (UniqueName: \"kubernetes.io/projected/4ff778a1-143c-471d-a387-3897770b9502-kube-api-access-vn28c\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysconfig\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-etc-kubernetes\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-modprobe-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142828 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-dbus\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-system-cni-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142920 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-hosts-file\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.142976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-sys-fs\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhvt\" (UniqueName: \"kubernetes.io/projected/0fda59f9-289b-431d-ba24-c291169fe1ae-kube-api-access-gdhvt\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25f0afe-e3c3-4ded-ab76-c98083607e36-iptables-alerter-script\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-sys-fs\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-bin\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-socket-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-tmp-dir\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63419a67-aaca-408d-bcfa-7c302b749c9d-serviceca\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-bin\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.143299 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-log-socket\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-netns\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-netns\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-etc-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-ovn\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff778a1-143c-471d-a387-3897770b9502-ovn-node-metrics-cert\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-registration-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-host\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphfn\" (UniqueName: \"kubernetes.io/projected/3d090b9a-abb1-48c3-9a43-415165ea4c04-kube-api-access-pphfn\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-cnibin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-socket-dir-parent\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsz8\" (UniqueName: \"kubernetes.io/projected/63419a67-aaca-408d-bcfa-7c302b749c9d-kube-api-access-nzsz8\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-socket-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lts\" (UniqueName: \"kubernetes.io/projected/1f47fdf3-f591-4184-abff-d4670f404937-kube-api-access-w2lts\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10f64c92-95d2-4fc4-add2-d174ec7fe24f-konnectivity-ca\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-tuned\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.144170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143611 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-netns\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143573 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-k8s-cni-cncf-io\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-ovn\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143702 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25f0afe-e3c3-4ded-ab76-c98083607e36-iptables-alerter-script\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143731 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-etc-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143735 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143729 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-log-socket\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143802 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-cnibin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-socket-dir-parent\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143814 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-registration-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-host\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-k8s-cni-cncf-io\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-bin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-var-lib-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143968 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.145071 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-run\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-var-lib-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.143993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-bin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8cv\" (UniqueName: \"kubernetes.io/projected/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kube-api-access-gj8cv\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-cnibin\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144064 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-run\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-kubernetes\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-sys\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144110 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-cnibin\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25f0afe-e3c3-4ded-ab76-c98083607e36-host-slash\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-cni-binary-copy\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63419a67-aaca-408d-bcfa-7c302b749c9d-serviceca\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144192 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-sys\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-multus\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144215 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25f0afe-e3c3-4ded-ab76-c98083607e36-host-slash\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144174 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-kubernetes\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-kubelet\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.145900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144257 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-multus\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-kubelet\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-system-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-multus-certs\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-system-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-netns\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-multus-certs\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.144456 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-script-lib\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-systemd\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.144586 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:55:56.644563348 +0000 UTC m=+3.193369985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:56.146718 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2k2\" (UniqueName: \"kubernetes.io/projected/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-kube-api-access-4d2k2\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-systemd\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-hostroot\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-kubelet-config\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144677 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-hostroot\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-config\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-conf\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-kubelet-config\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144755 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-lib-modules\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63419a67-aaca-408d-bcfa-7c302b749c9d-host\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144780 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-cni-binary-copy\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-kubelet\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-var-lib-kubelet\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144915 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-lib-modules\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63419a67-aaca-408d-bcfa-7c302b749c9d-host\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.144924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-slash\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-node-log\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145022 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-var-lib-kubelet\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-node-log\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-kubelet\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145078 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-conf\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-netd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-script-lib\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-device-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145128 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-slash\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-device-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577tv\" (UniqueName: \"kubernetes.io/projected/e951e6b3-30a9-4181-912e-1a40f1725862-kube-api-access-577tv\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145266 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-netd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-systemd-units\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-systemd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:56.147761 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-conf-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-multus-daemon-config\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145421 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-systemd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-systemd-units\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10f64c92-95d2-4fc4-add2-d174ec7fe24f-agent-certs\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.145496 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-env-overrides\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-conf-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-tmp\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnx6j\" (UniqueName: \"kubernetes.io/projected/d25f0afe-e3c3-4ded-ab76-c98083607e36-kube-api-access-qnx6j\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.145633 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:55:56.64561404 +0000 UTC m=+3.194420671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.145863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.146077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-env-overrides\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148411 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.146093 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-config\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148902 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.146151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.148902 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.146511 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-multus-daemon-config\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.148902 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.148077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-tmp\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.148902 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.148120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-tuned\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.148902 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.148172 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff778a1-143c-471d-a387-3897770b9502-ovn-node-metrics-cert\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.148902 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.148540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10f64c92-95d2-4fc4-add2-d174ec7fe24f-agent-certs\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.209125 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.209102 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:56.209125 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.209125 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:56.209338 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.209142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2k2\" (UniqueName: \"kubernetes.io/projected/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-kube-api-access-4d2k2\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.209338 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.209170 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:56.209338 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.209202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhvt\" (UniqueName: \"kubernetes.io/projected/0fda59f9-289b-431d-ba24-c291169fe1ae-kube-api-access-gdhvt\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:56.209338 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.209260 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:55:56.709242329 +0000 UTC m=+3.258048962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:56.209799 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.209779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsz8\" (UniqueName: \"kubernetes.io/projected/63419a67-aaca-408d-bcfa-7c302b749c9d-kube-api-access-nzsz8\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.210915 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.210888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8cv\" (UniqueName: \"kubernetes.io/projected/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kube-api-access-gj8cv\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.211971 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.211948 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577tv\" (UniqueName: \"kubernetes.io/projected/e951e6b3-30a9-4181-912e-1a40f1725862-kube-api-access-577tv\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.212369 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.212348 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn28c\" (UniqueName: \"kubernetes.io/projected/4ff778a1-143c-471d-a387-3897770b9502-kube-api-access-vn28c\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.212876 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.212853 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphfn\" (UniqueName: \"kubernetes.io/projected/3d090b9a-abb1-48c3-9a43-415165ea4c04-kube-api-access-pphfn\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.213254 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.213232 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnx6j\" (UniqueName: \"kubernetes.io/projected/d25f0afe-e3c3-4ded-ab76-c98083607e36-kube-api-access-qnx6j\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.213305 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.213272 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lts\" (UniqueName: \"kubernetes.io/projected/1f47fdf3-f591-4184-abff-d4670f404937-kube-api-access-w2lts\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.260392 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.260366 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:55:56.267057 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.267032 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:55:56.271017 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.270993 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25f0afe_e3c3_4ded_ab76_c98083607e36.slice/crio-9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00 WatchSource:0}: Error finding container 9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00: Status 404 returned error can't find the container with id 9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00 Oct 31 00:55:56.274652 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.274634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-649pf" Oct 31 00:55:56.275804 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.275786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zdntg" Oct 31 00:55:56.276462 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.276439 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f47fdf3_f591_4184_abff_d4670f404937.slice/crio-b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1 WatchSource:0}: Error finding container b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1: Status 404 returned error can't find the container with id b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1 Oct 31 00:55:56.285088 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.285069 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3537b0f9_a07f_423d_a9e2_8a3abc3c3916.slice/crio-21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb WatchSource:0}: Error finding container 21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb: Status 404 returned error can't find the container with id 21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb Oct 31 00:55:56.287902 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.287882 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode951e6b3_30a9_4181_912e_1a40f1725862.slice/crio-4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05 WatchSource:0}: Error finding container 4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05: Status 404 returned error can't find the container with id 4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05 Oct 31 00:55:56.291175 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.291158 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:55:56.293292 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.293273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:55:56.296081 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.296066 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:55:56.298514 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.298481 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:55:56.300930 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.300913 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:55:56.301452 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.301433 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63419a67_aaca_408d_bcfa_7c302b749c9d.slice/crio-52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa WatchSource:0}: Error finding container 52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa: Status 404 returned error can't find the container with id 52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa Oct 31 00:55:56.304997 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.304900 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee1f49a_b7d6_46f9_b402_86429c1be8e2.slice/crio-f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045 WatchSource:0}: Error finding container f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045: Status 404 returned error can't find the container with id f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045 Oct 31 00:55:56.311448 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.311420 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff778a1_143c_471d_a387_3897770b9502.slice/crio-373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b WatchSource:0}: Error finding container 373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b: Status 404 returned error can't find the container with id 373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b Oct 31 00:55:56.314734 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.314712 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f64c92_95d2_4fc4_add2_d174ec7fe24f.slice/crio-55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141 WatchSource:0}: Error finding container 55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141: Status 404 returned error can't find the container with id 55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141 Oct 31 00:55:56.317565 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:55:56.317545 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d090b9a_abb1_48c3_9a43_415165ea4c04.slice/crio-a1c3f9b8cb7220f40bd558103c7d639c1652aa2eec2dd8d7612c529ca2c131ff WatchSource:0}: Error finding container a1c3f9b8cb7220f40bd558103c7d639c1652aa2eec2dd8d7612c529ca2c131ff: Status 404 returned error can't find the container with id a1c3f9b8cb7220f40bd558103c7d639c1652aa2eec2dd8d7612c529ca2c131ff Oct 31 00:55:56.650673 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.650232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:56.650673 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.650377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:56.650673 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.650434 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:56.650673 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.650505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:55:57.650482309 +0000 UTC m=+4.199288944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:56.650673 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.650535 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:56.650673 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.650604 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:55:57.650587072 +0000 UTC m=+4.199393693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:56.750743 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.750708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:56.750952 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.750855 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:56.750952 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.750894 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:56.750952 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.750906 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:56.751100 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:56.750963 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:55:57.750945939 +0000 UTC m=+4.299752555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:56.987371 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.987080 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lttht" event={"ID":"10f64c92-95d2-4fc4-add2-d174ec7fe24f","Type":"ContainerStarted","Data":"55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141"} Oct 31 00:55:56.990637 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.990606 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zdntg" event={"ID":"e951e6b3-30a9-4181-912e-1a40f1725862","Type":"ContainerStarted","Data":"4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05"} Oct 31 00:55:56.992120 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.992082 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-649pf" event={"ID":"3537b0f9-a07f-423d-a9e2-8a3abc3c3916","Type":"ContainerStarted","Data":"21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb"} Oct 31 00:55:56.993398 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.993376 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerStarted","Data":"b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1"} Oct 31 00:55:56.994633 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.994602 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gbk88" event={"ID":"3d090b9a-abb1-48c3-9a43-415165ea4c04","Type":"ContainerStarted","Data":"a1c3f9b8cb7220f40bd558103c7d639c1652aa2eec2dd8d7612c529ca2c131ff"} Oct 31 00:55:56.996002 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.995959 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b"} Oct 31 00:55:56.997179 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.997157 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" event={"ID":"fee1f49a-b7d6-46f9-b402-86429c1be8e2","Type":"ContainerStarted","Data":"f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045"} Oct 31 00:55:56.998502 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.998476 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrqjp" event={"ID":"63419a67-aaca-408d-bcfa-7c302b749c9d","Type":"ContainerStarted","Data":"52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa"} Oct 31 00:55:56.999630 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:56.999610 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzlwl" event={"ID":"d25f0afe-e3c3-4ded-ab76-c98083607e36","Type":"ContainerStarted","Data":"9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00"} Oct 31 00:55:57.051094 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.051024 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:56 +0000 UTC" deadline="2028-06-22 02:34:47.782397321 +0000 UTC" Oct 31 00:55:57.051094 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.051086 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h38m50.731315699s" Oct 31 00:55:57.658578 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.658540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:57.658770 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.658625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:57.658861 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.658773 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:57.658861 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.658836 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:55:59.658816789 +0000 UTC m=+6.207623422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:57.659020 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.658953 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:57.659020 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.658997 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:55:59.658982561 +0000 UTC m=+6.207789191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:57.759353 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.759272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:57.759523 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.759450 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:57.759523 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.759471 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:57.759523 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.759482 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:57.759687 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.759536 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:55:59.759520129 +0000 UTC m=+6.308326761 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:57.982162 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.980043 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:57.982162 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.980190 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:55:57.982162 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.980276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:57.982162 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.980349 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:55:57.982162 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:57.980402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:57.982162 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:57.980469 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:55:58.051682 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:58.051576 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:56 +0000 UTC" deadline="2028-05-16 05:33:19.429035317 +0000 UTC" Oct 31 00:55:58.051682 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:58.051614 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13516h37m21.377426262s" Oct 31 00:55:59.025689 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.025312 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"e548c58b36505791ab3955482325fe49","Type":"ContainerStarted","Data":"e7437648e7b6143c4be811f5ccaf8151d8202903e98f20827e44d0b731945fc9"} Oct 31 00:55:59.676254 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.676221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:59.676707 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.676291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:59.676707 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.676366 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:59.676707 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.676420 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:59.676707 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.676424 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:56:03.676406141 +0000 UTC m=+10.225212768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:55:59.676707 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.676483 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:56:03.676467528 +0000 UTC m=+10.225274145 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:55:59.777281 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.777020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:59.777281 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.777161 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:55:59.777281 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.777184 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:55:59.777281 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.777196 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:59.777281 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.777248 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:56:03.777230716 +0000 UTC m=+10.326037333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:55:59.978973 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.978499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:55:59.978973 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.978614 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:55:59.978973 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.978715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:55:59.978973 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.978809 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:55:59.978973 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:55:59.978885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:55:59.978973 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:55:59.978945 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:00.034809 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:00.034626 2578 generic.go:356] "Generic (PLEG): container finished" podID="7b884c31d96cb5a9a71220f2bda573a2" containerID="f0b3c575252f55f2d02bdb3900650d7693c8f478ef58ea543810b1d982f556b1" exitCode=0 Oct 31 00:56:00.034809 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:00.034727 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"7b884c31d96cb5a9a71220f2bda573a2","Type":"ContainerDied","Data":"f0b3c575252f55f2d02bdb3900650d7693c8f478ef58ea543810b1d982f556b1"} Oct 31 00:56:00.040252 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:00.040209 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="34b7f8ed05253d37880c287ba7c138eafa6eff49b7562115a65742c9dde97e50" exitCode=0 Oct 31 00:56:00.041190 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:00.041166 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerDied","Data":"34b7f8ed05253d37880c287ba7c138eafa6eff49b7562115a65742c9dde97e50"} Oct 31 00:56:00.152825 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:00.152710 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" podStartSLOduration=5.15268807 podStartE2EDuration="5.15268807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:59.143173487 +0000 UTC m=+5.691980145" watchObservedRunningTime="2026-10-31 00:56:00.15268807 +0000 UTC m=+6.701494705" Oct 31 00:56:01.052895 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.052253 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"7b884c31d96cb5a9a71220f2bda573a2","Type":"ContainerStarted","Data":"4413cb6cb647eb0c70a671b52dd4e8349827d9364df119dd3afdef30ce8a27a5"} Oct 31 00:56:01.057642 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.057209 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lttht" event={"ID":"10f64c92-95d2-4fc4-add2-d174ec7fe24f","Type":"ContainerStarted","Data":"64cc18e614175d064a8a0f774ddde64eda2cd90493bd06e5832642172c019a43"} Oct 31 00:56:01.172919 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.172780 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" podStartSLOduration=6.172760359 podStartE2EDuration="6.172760359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:01.17240249 +0000 UTC m=+7.721209128" watchObservedRunningTime="2026-10-31 00:56:01.172760359 +0000 UTC m=+7.721566996" Oct 31 00:56:01.288900 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.288200 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lttht" podStartSLOduration=2.767138142 podStartE2EDuration="7.288181842s" totalImagesPullingTime="4.5210437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:01.287814807 +0000 UTC m=+7.836621443" watchObservedRunningTime="2026-10-31 00:56:01.288181842 +0000 UTC m=+7.836988477" Oct 31 00:56:01.769154 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.769125 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:56:01.770013 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.769648 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:56:01.981696 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.981269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:01.981696 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:01.981387 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:01.981696 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.981456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:01.981696 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:01.981532 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:01.981696 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:01.981582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:01.981696 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:01.981649 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:03.066149 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.066107 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" event={"ID":"fee1f49a-b7d6-46f9-b402-86429c1be8e2","Type":"ContainerStarted","Data":"5ca354274649c57b2901f55da10545ec8a3bbff9b1e89e07c3323e4501337720"} Oct 31 00:56:03.068668 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.068644 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:56:03.069492 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.069468 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrqjp" event={"ID":"63419a67-aaca-408d-bcfa-7c302b749c9d","Type":"ContainerStarted","Data":"8d8d0909c29a74293a6082e2d3f559d9fd52abef8027e8e9312f588b05c7e673"} Oct 31 00:56:03.185454 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.185252 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vrqjp" podStartSLOduration=3.465845148 podStartE2EDuration="9.18523205s" totalImagesPullingTime="5.719386902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:03.184133046 +0000 UTC m=+9.732939686" watchObservedRunningTime="2026-10-31 00:56:03.18523205 +0000 UTC m=+9.734038687" Oct 31 00:56:03.711484 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.711449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:03.711661 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.711526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:03.711661 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.711619 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:56:03.711661 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.711647 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:56:03.711831 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.711687 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:56:11.711667584 +0000 UTC m=+18.260474215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:56:03.711831 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.711713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:56:11.711695327 +0000 UTC m=+18.260501942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:56:03.811998 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.811962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:03.812174 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.812162 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:56:03.812240 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.812185 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:56:03.812240 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.812197 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:56:03.812340 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.812251 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:56:11.812233045 +0000 UTC m=+18.361039665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:56:03.984242 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.983655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:03.984242 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.983769 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:03.984242 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.983981 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:03.984242 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:03.983993 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:03.984242 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.984070 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:03.984242 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:03.984173 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:04.637711 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:04.637681 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:56:04.916657 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:04.916010 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:56:04.637705786Z","UUID":"bc2e6e82-5b45-428f-8f5a-e472412748bc","Handler":null,"Name":"","Endpoint":""} Oct 31 00:56:04.918639 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:04.918613 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:56:04.918746 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:04.918648 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:56:05.074347 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.074087 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zdntg" event={"ID":"e951e6b3-30a9-4181-912e-1a40f1725862","Type":"ContainerStarted","Data":"fe8676bbe6bf454fc17901b46d281f455663c4888bbb4184d2943568d926d84e"} Oct 31 00:56:05.076122 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.075709 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-649pf" event={"ID":"3537b0f9-a07f-423d-a9e2-8a3abc3c3916","Type":"ContainerStarted","Data":"b7f59d72d134becefd00212aa67057e507563a4f29ac0ba34d7cb76ecac9534f"} Oct 31 00:56:05.078014 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.077989 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="c400bf6a680156257533eb3f7d976ac66d89c2ea26d82cb1d3c2338ccb287558" exitCode=0 Oct 31 00:56:05.078139 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.078052 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerDied","Data":"c400bf6a680156257533eb3f7d976ac66d89c2ea26d82cb1d3c2338ccb287558"} Oct 31 00:56:05.079999 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.079966 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" event={"ID":"fee1f49a-b7d6-46f9-b402-86429c1be8e2","Type":"ContainerStarted","Data":"f2373b22f84eedeb701f846c7fba5d3bfe5773b03a54bf350c25e3fc16d79655"} Oct 31 00:56:05.310162 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.310113 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-649pf" podStartSLOduration=2.779623865 podStartE2EDuration="11.310098729s" totalImagesPullingTime="8.530474864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:05.309511814 +0000 UTC m=+11.858318469" watchObservedRunningTime="2026-10-31 00:56:05.310098729 +0000 UTC m=+11.858905365" Oct 31 00:56:05.310343 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.310186 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zdntg" podStartSLOduration=3.136016165 podStartE2EDuration="11.31018139s" totalImagesPullingTime="8.174165225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:05.190814938 +0000 UTC m=+11.739621573" watchObservedRunningTime="2026-10-31 00:56:05.31018139 +0000 UTC m=+11.858988025" Oct 31 00:56:05.979615 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.979580 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:05.980082 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.979594 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:05.980082 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:05.979817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:05.980082 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:05.979720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:05.980082 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:05.979705 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:05.980082 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:05.979913 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:06.086391 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:06.086352 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzlwl" event={"ID":"d25f0afe-e3c3-4ded-ab76-c98083607e36","Type":"ContainerStarted","Data":"00be91b1375d01a5908ed9c4aed288ec7697ef692d31e4f0b0f09878f062d1d0"} Oct 31 00:56:06.201845 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:06.201782 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nzlwl" podStartSLOduration=3.806402767 podStartE2EDuration="12.201767272s" totalImagesPullingTime="8.395364505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:06.201291036 +0000 UTC m=+12.750097672" watchObservedRunningTime="2026-10-31 00:56:06.201767272 +0000 UTC m=+12.750573906" Oct 31 00:56:07.089942 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.089910 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="6b3b487b8c1b6e6355e072675c11419fffe7bfdf71118fb3b33fa9193b2c0113" exitCode=0 Oct 31 00:56:07.090584 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.089985 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerDied","Data":"6b3b487b8c1b6e6355e072675c11419fffe7bfdf71118fb3b33fa9193b2c0113"} Oct 31 00:56:07.091552 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.091523 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gbk88" event={"ID":"3d090b9a-abb1-48c3-9a43-415165ea4c04","Type":"ContainerStarted","Data":"ca11c8f772d927f2686362089edca396fb7cfdf136581b7ae509113d55f61cc9"} Oct 31 00:56:07.093617 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.093596 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" event={"ID":"fee1f49a-b7d6-46f9-b402-86429c1be8e2","Type":"ContainerStarted","Data":"7fd9567dca0844cb77bccc2d7f7e3fdb8e097846c1663fb0f4b4fe2a8412917a"} Oct 31 00:56:07.331055 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.330992 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" podStartSLOduration=3.944425389 podStartE2EDuration="13.33096459s" totalImagesPullingTime="9.386539201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:07.330639203 +0000 UTC m=+13.879445844" watchObservedRunningTime="2026-10-31 00:56:07.33096459 +0000 UTC m=+13.879771226" Oct 31 00:56:07.979073 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.979037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:07.979217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.979084 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:07.979217 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:07.979047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:07.979217 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:07.979151 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:07.979331 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:07.979267 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:07.979380 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:07.979360 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:09.099256 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:09.099222 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="ca2e35e8d45c3b680ad98fd62f743af5763d0930945785179a3f793a04a2e1f8" exitCode=0 Oct 31 00:56:09.099712 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:09.099271 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerDied","Data":"ca2e35e8d45c3b680ad98fd62f743af5763d0930945785179a3f793a04a2e1f8"} Oct 31 00:56:09.320178 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:09.319905 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gbk88" podStartSLOduration=4.755610708 podStartE2EDuration="15.319886922s" totalImagesPullingTime="10.564276214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:07.454626254 +0000 UTC m=+14.003432889" watchObservedRunningTime="2026-10-31 00:56:09.319886922 +0000 UTC m=+15.868693553" Oct 31 00:56:09.979095 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:09.979060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:09.979277 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:09.979062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:09.979277 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:09.979164 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:09.979277 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:09.979248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:09.979277 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:09.979067 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:09.979447 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:09.979353 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:11.772227 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:11.772120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:11.772227 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:11.772215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:11.772691 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.772294 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:56:11.772691 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.772307 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:56:11.772691 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.772359 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:56:27.772344712 +0000 UTC m=+34.321151324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:56:11.772691 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.772382 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:56:27.772370204 +0000 UTC m=+34.321176822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:56:11.873171 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:11.873133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:11.873344 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.873327 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:56:11.873404 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.873351 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:56:11.873404 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.873364 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:56:11.873506 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.873421 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:56:27.873403213 +0000 UTC m=+34.422209841 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:56:11.978576 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:11.978542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:11.978576 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:11.978556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:11.978808 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.978681 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:11.979144 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.979111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:11.979291 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:11.979178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:11.979291 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:11.979246 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:12.109576 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:12.109541 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="7abb589564d9fff8b2d6a2db95f1a32c958eacd32ffd3501bafdbcf105681373" exitCode=0 Oct 31 00:56:12.109760 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:12.109599 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerDied","Data":"7abb589564d9fff8b2d6a2db95f1a32c958eacd32ffd3501bafdbcf105681373"} Oct 31 00:56:13.115396 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:13.115367 2578 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="10aaf6d46e7b0f9e66d0b3e1f846fd7b2255ff2f28a789519b147cc6f562802e" exitCode=0 Oct 31 00:56:13.115815 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:13.115403 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerDied","Data":"10aaf6d46e7b0f9e66d0b3e1f846fd7b2255ff2f28a789519b147cc6f562802e"} Oct 31 00:56:13.981288 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:13.981257 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:13.981288 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:13.981277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:13.981536 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:13.981360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:13.981536 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:13.981367 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:13.981536 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:13.981451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:13.981536 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:13.981509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:14.121459 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:14.121429 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" event={"ID":"1f47fdf3-f591-4184-abff-d4670f404937","Type":"ContainerStarted","Data":"ae26a0a6d4ca2cb6bbc66310c2af0500a5b34a672b683fd5b72e96130fb19f46"} Oct 31 00:56:14.270055 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:14.269947 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ztqdl" podStartSLOduration=6.909060389 podStartE2EDuration="20.269930067s" totalImagesPullingTime="13.360869678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:14.269714889 +0000 UTC m=+20.818521526" watchObservedRunningTime="2026-10-31 00:56:14.269930067 +0000 UTC m=+20.818736703" Oct 31 00:56:15.125809 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:15.125661 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"74b8804393abbeab572ce6c9955a609cfe1d83f9176be0aee237f10c6ffcd5b5"} Oct 31 00:56:15.126170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:15.125902 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"cf1b52e95f945677c2d5754ba84237cccf7628c449cfe3a9fb8c8da93aa6f9f5"} Oct 31 00:56:15.126170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:15.125937 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"fcc2e4183efb0592c000e08aa9441450806e72c62b15c9280f0c44708bcf9563"} Oct 31 00:56:15.981566 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:15.981533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:15.981566 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:15.981559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:15.981786 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:15.981643 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:15.981786 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:15.981657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:15.981786 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:15.981719 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:15.981913 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:15.981796 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:16.130494 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:16.130462 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"6798274fe7731633ef37a3c545e99c7ad95ce543d261ccbe6bdafe0d1fa0ffcf"} Oct 31 00:56:16.130852 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:16.130522 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"93dce1202e850f59ca0267b1c11974ed2fbbf96c857ceb133e1d4fc12d4fcc5b"} Oct 31 00:56:16.130852 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:16.130551 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"074039d15a1931739c018aa0cbf49f9f43a346b075cd30edaa14f70d5477ac62"} Oct 31 00:56:16.192107 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:16.192078 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:56:16.192207 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:16.192196 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:56:16.192742 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:16.192721 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:56:17.983544 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:17.983514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:17.984199 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:17.983601 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:17.984199 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:17.983692 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:17.984199 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:17.983695 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:17.984199 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:17.983762 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:17.984199 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:17.983876 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:18.138014 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:18.137983 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"41c51a4db55cd229d37c18f9a395fca1ce4f005c774c43719a9dea3a9248d9d1"} Oct 31 00:56:19.978611 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:19.978582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:19.979253 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:19.978582 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:19.979253 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:19.978686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:19.979253 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:19.978596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:19.979253 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:19.978759 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:19.979253 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:19.978852 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:20.145406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:20.145376 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" event={"ID":"4ff778a1-143c-471d-a387-3897770b9502","Type":"ContainerStarted","Data":"b12f5e54e60d5f2d655c892d4f79b44995b970d38df3cba8efe9d04e7468e67f"} Oct 31 00:56:20.147629 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:20.146905 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:56:20.147629 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:20.147036 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:56:20.163012 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:20.162986 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:56:20.163145 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:20.163084 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:56:20.422364 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:20.422312 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" podStartSLOduration=7.788063636 podStartE2EDuration="26.422298671s" totalImagesPullingTime="18.634235035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:20.371218695 +0000 UTC m=+26.920025331" watchObservedRunningTime="2026-10-31 00:56:20.422298671 +0000 UTC m=+26.971105329" Oct 31 00:56:21.131102 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:21.131069 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:56:21.982696 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:21.982433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:21.982696 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:21.982638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:21.982917 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:21.982596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:21.982917 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:21.982739 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:21.982917 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:21.982798 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:21.982917 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:21.982878 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:22.084333 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:22.084297 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-blwrh"] Oct 31 00:56:22.084869 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:22.084839 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pgqj5"] Oct 31 00:56:22.085360 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:22.085340 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-48ddr"] Oct 31 00:56:22.150136 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:22.150106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:22.150511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:22.150148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:22.150511 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:22.150212 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:22.150511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:22.150234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:22.150511 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:22.150302 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:22.150511 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:22.150440 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:23.979459 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:23.979424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:23.979898 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:23.979547 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:23.979898 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:23.979595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:23.979898 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:23.979653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:23.979898 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:23.979748 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:23.979898 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:23.979833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:25.978833 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:25.978795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:25.978833 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:25.978811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:25.979376 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:25.978929 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:25.979376 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:25.978972 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:25.979376 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:25.979020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:25.979376 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:25.979084 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:27.778321 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:27.778239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:27.778321 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:27.778298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:27.778782 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.778391 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:56:27.778782 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.778391 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 00:56:27.778782 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.778454 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs podName:0fda59f9-289b-431d-ba24-c291169fe1ae nodeName:}" failed. No retries permitted until 2026-10-31 00:56:59.778435113 +0000 UTC m=+66.327241725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs") pod "network-metrics-daemon-pgqj5" (UID: "0fda59f9-289b-431d-ba24-c291169fe1ae") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 00:56:27.778782 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.778470 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret podName:66bb82ee-c3c0-4719-8a69-04ac4ce085cd nodeName:}" failed. No retries permitted until 2026-10-31 00:56:59.77846156 +0000 UTC m=+66.327268173 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret") pod "global-pull-secret-syncer-48ddr" (UID: "66bb82ee-c3c0-4719-8a69-04ac4ce085cd") : object "kube-system"/"original-pull-secret" not registered Oct 31 00:56:27.878682 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:27.878652 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:27.878821 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.878785 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 00:56:27.878821 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.878798 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 00:56:27.878821 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.878806 2578 projected.go:196] Error preparing data for projected volume kube-api-access-58psx for pod openshift-network-diagnostics/network-check-target-blwrh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:56:27.878949 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.878853 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx podName:d8189bdc-c813-4a8d-a1e0-052a4415d4ba nodeName:}" failed. No retries permitted until 2026-10-31 00:56:59.878837758 +0000 UTC m=+66.427644389 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-58psx" (UniqueName: "kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx") pod "network-check-target-blwrh" (UID: "d8189bdc-c813-4a8d-a1e0-052a4415d4ba") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 00:56:27.978531 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:27.978501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:27.978659 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:27.978555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:27.978659 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:27.978573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:27.978659 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.978634 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:27.978766 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.978703 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:27.978810 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:27.978778 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:29.978736 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:29.978712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:29.978736 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:29.978743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:29.979197 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:29.978710 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:29.979197 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:29.978817 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:29.979197 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:29.978919 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:29.979197 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:29.979026 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:30.915800 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:30.915768 2578 ???:1] "http: TLS handshake error from 10.0.139.174:44648: EOF" Oct 31 00:56:30.962888 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:30.962853 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-649pf_3537b0f9-a07f-423d-a9e2-8a3abc3c3916/dns-node-resolver/0.log" Oct 31 00:56:30.963105 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:30.963069 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lttht_10f64c92-95d2-4fc4-add2-d174ec7fe24f/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:31.227384 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:31.227269 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lttht_10f64c92-95d2-4fc4-add2-d174ec7fe24f/konnectivity-agent/0.log" limit=1024 Oct 31 00:56:31.978488 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:31.978415 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:31.978488 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:31.978459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:31.978649 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:31.978582 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:31.978649 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:31.978610 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:31.978649 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:31.978644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:31.978740 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:31.978717 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:32.178499 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:32.178475 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrqjp_63419a67-aaca-408d-bcfa-7c302b749c9d/node-ca/0.log" Oct 31 00:56:33.979005 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:33.978976 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:33.979355 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:33.979098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:33.979355 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:33.979116 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:33.979355 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:33.979207 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:33.979355 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:33.979241 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:33.979355 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:33.979322 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:35.981309 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:35.981275 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:35.981888 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:35.981391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:35.981888 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:35.981417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:35.981888 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:35.981408 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:35.981888 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:35.981490 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:35.981888 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:35.981558 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:37.978960 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:37.978926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:37.979458 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:37.979020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:37.979458 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:37.979049 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:37.979458 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:37.979060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:37.979458 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:37.979123 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:37.979458 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:37.979189 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:39.978856 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:39.978653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:39.979302 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:39.978653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:39.979302 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:39.978966 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:39.979302 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:39.979018 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:39.979302 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:39.978668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:39.979302 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:39.979112 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:41.979052 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:41.979022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:41.979480 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:41.979060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:41.979480 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:41.979023 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:41.979480 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:41.979149 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:41.979480 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:41.979217 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:41.979480 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:41.979312 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:43.978953 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:43.978926 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:43.979352 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:43.979054 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:43.979352 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:43.979090 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:43.979352 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:43.979134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:43.979352 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:43.979209 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:43.979352 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:43.979317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:45.979038 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:45.979010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:45.979528 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:45.979016 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:45.979528 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:45.979110 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:45.979528 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:45.979211 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:45.979528 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:45.979021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:45.979528 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:45.979298 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:47.979066 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:47.979024 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:47.979066 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:47.979042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:47.979584 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:47.979144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:47.979584 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:47.979155 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-blwrh" podUID="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" Oct 31 00:56:47.979584 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:47.979284 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pgqj5" podUID="0fda59f9-289b-431d-ba24-c291169fe1ae" Oct 31 00:56:47.979584 ip-10-0-136-249 kubenswrapper[2578]: E1031 00:56:47.979389 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48ddr" podUID="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" Oct 31 00:56:49.219443 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.219423 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeReady" Oct 31 00:56:49.219767 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.219529 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 00:56:49.380233 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.380204 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gp5rq"] Oct 31 00:56:49.381836 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.381822 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.395089 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.395070 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c7pm5"] Oct 31 00:56:49.396536 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.396524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.425579 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.425530 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wklrq"] Oct 31 00:56:49.427356 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.427343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.443031 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.443013 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:56:49.443278 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.443264 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:56:49.443394 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.443383 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk5g6\"" Oct 31 00:56:49.443427 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.443394 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:56:49.446682 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.446665 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gp5rq"] Oct 31 00:56:49.459971 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.459945 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c7pm5"] Oct 31 00:56:49.492014 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.491987 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wklrq"] Oct 31 00:56:49.518012 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.517983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbaba560-5774-4a19-afde-447fb6548557-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.518129 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.518016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbaba560-5774-4a19-afde-447fb6548557-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.518129 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.518105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl4lx\" (UniqueName: \"kubernetes.io/projected/cbaba560-5774-4a19-afde-447fb6548557-kube-api-access-hl4lx\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.518198 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.518184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01262e31-9355-45a2-adc6-2f7889c4fe8e-cert\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.518232 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.518210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fcnq\" (UniqueName: \"kubernetes.io/projected/01262e31-9355-45a2-adc6-2f7889c4fe8e-kube-api-access-2fcnq\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.518271 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.518258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbaba560-5774-4a19-afde-447fb6548557-data-volume\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.518304 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.518291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbaba560-5774-4a19-afde-447fb6548557-crio-socket\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.543558 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.543535 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:56:49.543665 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.543653 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:56:49.543760 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.543748 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:56:49.543984 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.543967 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:56:49.544111 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.544099 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:56:49.544145 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.544098 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rf2ln\"" Oct 31 00:56:49.544251 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.544238 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:56:49.544283 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.544244 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m5b2l\"" Oct 31 00:56:49.619003 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.618981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbaba560-5774-4a19-afde-447fb6548557-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619100 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5spv\" (UniqueName: \"kubernetes.io/projected/41b41813-6309-49be-bd2c-76b0820899af-kube-api-access-n5spv\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.619100 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4lx\" (UniqueName: \"kubernetes.io/projected/cbaba560-5774-4a19-afde-447fb6548557-kube-api-access-hl4lx\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619100 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01262e31-9355-45a2-adc6-2f7889c4fe8e-cert\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.619385 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcnq\" (UniqueName: \"kubernetes.io/projected/01262e31-9355-45a2-adc6-2f7889c4fe8e-kube-api-access-2fcnq\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.619465 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41b41813-6309-49be-bd2c-76b0820899af-config-volume\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.619465 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbaba560-5774-4a19-afde-447fb6548557-data-volume\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619591 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619568 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b41813-6309-49be-bd2c-76b0820899af-tmp-dir\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.619647 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbaba560-5774-4a19-afde-447fb6548557-crio-socket\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619647 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41b41813-6309-49be-bd2c-76b0820899af-metrics-tls\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.619748 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619713 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbaba560-5774-4a19-afde-447fb6548557-data-volume\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619790 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619743 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbaba560-5774-4a19-afde-447fb6548557-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619831 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619796 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbaba560-5774-4a19-afde-447fb6548557-crio-socket\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.619831 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.619796 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbaba560-5774-4a19-afde-447fb6548557-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.623351 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.623330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbaba560-5774-4a19-afde-447fb6548557-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.623430 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.623373 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01262e31-9355-45a2-adc6-2f7889c4fe8e-cert\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.681174 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.681110 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4lx\" (UniqueName: \"kubernetes.io/projected/cbaba560-5774-4a19-afde-447fb6548557-kube-api-access-hl4lx\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.682446 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.682425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcnq\" (UniqueName: \"kubernetes.io/projected/01262e31-9355-45a2-adc6-2f7889c4fe8e-kube-api-access-2fcnq\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.692195 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.692177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:56:49.707655 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.707628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:56:49.720461 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.720435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5spv\" (UniqueName: \"kubernetes.io/projected/41b41813-6309-49be-bd2c-76b0820899af-kube-api-access-n5spv\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.720554 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.720522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41b41813-6309-49be-bd2c-76b0820899af-config-volume\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.720612 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.720592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b41813-6309-49be-bd2c-76b0820899af-tmp-dir\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.720664 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.720628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41b41813-6309-49be-bd2c-76b0820899af-metrics-tls\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.720947 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.720925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b41813-6309-49be-bd2c-76b0820899af-tmp-dir\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.721073 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.721056 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41b41813-6309-49be-bd2c-76b0820899af-config-volume\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.723170 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.723141 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41b41813-6309-49be-bd2c-76b0820899af-metrics-tls\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.780401 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.780374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5spv\" (UniqueName: \"kubernetes.io/projected/41b41813-6309-49be-bd2c-76b0820899af-kube-api-access-n5spv\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:49.978917 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.978829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:49.978917 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.978829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:49.979109 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:49.978949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:50.032352 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032328 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:56:50.032352 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032312 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:56:50.032541 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032484 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gjj6f\"" Oct 31 00:56:50.032597 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032564 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:56:50.032638 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032599 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hnbxm\"" Oct 31 00:56:50.032638 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032597 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:56:50.032638 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.032611 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxv7v\"" Oct 31 00:56:50.036701 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.036689 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:50.072122 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.072048 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gp5rq"] Oct 31 00:56:50.076955 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:56:50.076928 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01262e31_9355_45a2_adc6_2f7889c4fe8e.slice/crio-380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57 WatchSource:0}: Error finding container 380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57: Status 404 returned error can't find the container with id 380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57 Oct 31 00:56:50.128406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.128381 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c7pm5"] Oct 31 00:56:50.131019 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:56:50.130996 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbaba560_5774_4a19_afde_447fb6548557.slice/crio-4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b WatchSource:0}: Error finding container 4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b: Status 404 returned error can't find the container with id 4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b Oct 31 00:56:50.211370 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.209727 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c7pm5" event={"ID":"cbaba560-5774-4a19-afde-447fb6548557","Type":"ContainerStarted","Data":"8b00123e198f2d58a7ba3a5597db7671ea2c446e99c45eb92f1cbada29b3bdfc"} Oct 31 00:56:50.211370 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.209790 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c7pm5" event={"ID":"cbaba560-5774-4a19-afde-447fb6548557","Type":"ContainerStarted","Data":"4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b"} Oct 31 00:56:50.212957 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.212932 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gp5rq" event={"ID":"01262e31-9355-45a2-adc6-2f7889c4fe8e","Type":"ContainerStarted","Data":"380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57"} Oct 31 00:56:50.406694 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:50.406674 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wklrq"] Oct 31 00:56:50.408631 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:56:50.408603 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b41813_6309_49be_bd2c_76b0820899af.slice/crio-6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241 WatchSource:0}: Error finding container 6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241: Status 404 returned error can't find the container with id 6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241 Oct 31 00:56:51.216064 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:51.216023 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wklrq" event={"ID":"41b41813-6309-49be-bd2c-76b0820899af","Type":"ContainerStarted","Data":"6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241"} Oct 31 00:56:52.163699 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:52.163673 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:56:52.221164 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:52.221131 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c7pm5" event={"ID":"cbaba560-5774-4a19-afde-447fb6548557","Type":"ContainerStarted","Data":"66905661d17b58ea09d65114dfbef14d6abd932d936ad52b26c7c77dc300f72c"} Oct 31 00:56:53.226903 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.226839 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wklrq" event={"ID":"41b41813-6309-49be-bd2c-76b0820899af","Type":"ContainerStarted","Data":"6b277e1087e972a3184751c2ad0e1b9957526131663c14c16110cec3d0628b01"} Oct 31 00:56:53.227351 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.226960 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wklrq" event={"ID":"41b41813-6309-49be-bd2c-76b0820899af","Type":"ContainerStarted","Data":"d60f6375f1b397273826515f177579b6d91a17630650b1fbea07a52d44f51e97"} Oct 31 00:56:53.227351 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.227003 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wklrq" Oct 31 00:56:53.244143 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.244110 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l72fc"] Oct 31 00:56:53.246563 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.246543 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.247625 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.247597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.247730 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.247699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2b8r\" (UniqueName: \"kubernetes.io/projected/64d66bc1-aa19-496e-82c5-ec3bedaa9943-kube-api-access-h2b8r\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.247790 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.247731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64d66bc1-aa19-496e-82c5-ec3bedaa9943-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.247842 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.247818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.307093 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.307063 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:56:53.307295 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.307146 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:56:53.307295 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.307275 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:56:53.308287 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.308216 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l72fc"] Oct 31 00:56:53.330551 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.330517 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 00:56:53.330692 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.330627 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 00:56:53.330692 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.330673 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4fw5b\"" Oct 31 00:56:53.348369 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.348338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b8r\" (UniqueName: \"kubernetes.io/projected/64d66bc1-aa19-496e-82c5-ec3bedaa9943-kube-api-access-h2b8r\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.348517 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.348377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64d66bc1-aa19-496e-82c5-ec3bedaa9943-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.348517 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.348443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.348517 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.348488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.349650 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.349417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64d66bc1-aa19-496e-82c5-ec3bedaa9943-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.351577 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.351552 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.351769 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.351587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.406734 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.406706 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b8r\" (UniqueName: \"kubernetes.io/projected/64d66bc1-aa19-496e-82c5-ec3bedaa9943-kube-api-access-h2b8r\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.480941 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.480820 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wklrq" podStartSLOduration=2.364233447 podStartE2EDuration="4.480801571s" totalImagesPullingTime="2.116568124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:53.360660493 +0000 UTC m=+59.909467129" watchObservedRunningTime="2026-10-31 00:56:53.480801571 +0000 UTC m=+60.029608203" Oct 31 00:56:53.560808 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.560768 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:56:53.929088 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:53.929061 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l72fc"] Oct 31 00:56:53.931595 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:56:53.931570 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d66bc1_aa19_496e_82c5_ec3bedaa9943.slice/crio-9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e WatchSource:0}: Error finding container 9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e: Status 404 returned error can't find the container with id 9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e Oct 31 00:56:54.231165 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:54.231067 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gp5rq" event={"ID":"01262e31-9355-45a2-adc6-2f7889c4fe8e","Type":"ContainerStarted","Data":"bfe3f3371f0afdf022aad3d676175cc36cfd1b5631539e59967f18a8b7a5e9ae"} Oct 31 00:56:54.232275 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:54.232247 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" event={"ID":"64d66bc1-aa19-496e-82c5-ec3bedaa9943","Type":"ContainerStarted","Data":"9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e"} Oct 31 00:56:54.234318 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:54.234294 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c7pm5" event={"ID":"cbaba560-5774-4a19-afde-447fb6548557","Type":"ContainerStarted","Data":"48268519be7da3f0a93e2d171e23260a16304e2626e5646ba37be2ab12001ce4"} Oct 31 00:56:54.466357 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:54.466300 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gp5rq" podStartSLOduration=2.055030579 podStartE2EDuration="5.466285299s" totalImagesPullingTime="3.41125472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:54.347740991 +0000 UTC m=+60.896547646" watchObservedRunningTime="2026-10-31 00:56:54.466285299 +0000 UTC m=+61.015091934" Oct 31 00:56:54.466764 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:54.466744 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c7pm5" podStartSLOduration=2.387451307 podStartE2EDuration="5.466736823s" totalImagesPullingTime="3.079285516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:54.466065478 +0000 UTC m=+61.014872132" watchObservedRunningTime="2026-10-31 00:56:54.466736823 +0000 UTC m=+61.015543458" Oct 31 00:56:55.170148 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.170115 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6444d66d-l4zqt"] Oct 31 00:56:55.171904 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.171879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.260678 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.260651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.261135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.260722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.261135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.260790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.261135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.260885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.261135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.260983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.261135 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.261031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.261389 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.261141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.276156 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276126 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:56:55.276156 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276123 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:56:55.276332 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276179 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:56:55.276379 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276337 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:56:55.276409 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276390 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:56:55.276409 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276394 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r7zp6\"" Oct 31 00:56:55.276559 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276538 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:56:55.276681 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.276660 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:56:55.277399 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.277380 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6444d66d-l4zqt"] Oct 31 00:56:55.361525 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.361675 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.361675 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.361675 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.361675 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.361897 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.361897 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.361718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.362272 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.362247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.362598 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.362547 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.362767 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.362748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.362837 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.362807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.364260 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.364241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.364450 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.364430 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.421289 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.421227 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.484825 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.484795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:56:55.861786 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:55.861759 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6444d66d-l4zqt"] Oct 31 00:56:55.863883 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:56:55.863838 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2593841_e6ff_4bce_a8d8_a8161a7b895c.slice/crio-c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db WatchSource:0}: Error finding container c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db: Status 404 returned error can't find the container with id c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db Oct 31 00:56:56.242381 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:56.242345 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6444d66d-l4zqt" event={"ID":"b2593841-e6ff-4bce-a8d8-a8161a7b895c","Type":"ContainerStarted","Data":"c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db"} Oct 31 00:56:56.244066 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:56.244043 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" event={"ID":"64d66bc1-aa19-496e-82c5-ec3bedaa9943","Type":"ContainerStarted","Data":"c60a000b9f10698cb9d3fbb186279245d650980dd162e44c97930a1a7f15bb4a"} Oct 31 00:56:56.244199 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:56.244081 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" event={"ID":"64d66bc1-aa19-496e-82c5-ec3bedaa9943","Type":"ContainerStarted","Data":"d31b7bf0259ccbc5c84979bdbcc6c279fa3d560635f71c7e912078b98eceef42"} Oct 31 00:56:56.362508 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:56.362458 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" podStartSLOduration=1.609553741 podStartE2EDuration="3.362445368s" totalImagesPullingTime="1.752891627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:56.361415127 +0000 UTC m=+62.910221761" watchObservedRunningTime="2026-10-31 00:56:56.362445368 +0000 UTC m=+62.911252002" Oct 31 00:56:59.257211 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.257134 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6444d66d-l4zqt" event={"ID":"b2593841-e6ff-4bce-a8d8-a8161a7b895c","Type":"ContainerStarted","Data":"ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98"} Oct 31 00:56:59.374792 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.374742 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c6444d66d-l4zqt" podStartSLOduration=1.248774203 podStartE2EDuration="4.374728053s" totalImagesPullingTime="3.12595385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:59.374329711 +0000 UTC m=+65.923136369" watchObservedRunningTime="2026-10-31 00:56:59.374728053 +0000 UTC m=+65.923534688" Oct 31 00:56:59.697095 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.697065 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7"] Oct 31 00:56:59.699077 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.699061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.745377 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.745350 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs"] Oct 31 00:56:59.747810 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.747786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.795225 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795192 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:56:59.795225 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795201 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:56:59.795225 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795227 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tzg24\"" Oct 31 00:56:59.795511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.795511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:59.795511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.795511 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.795669 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.795669 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.795769 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.795769 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74t85\" (UniqueName: \"kubernetes.io/projected/fc981b83-c624-4524-95c0-950ec6a283ee-kube-api-access-74t85\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.795923 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq5q4\" (UniqueName: \"kubernetes.io/projected/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-api-access-kq5q4\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.795997 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.795946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:59.796051 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.796008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.796102 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.796046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc981b83-c624-4524-95c0-950ec6a283ee-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.796318 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.796300 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7"] Oct 31 00:56:59.803832 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.803818 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:56:59.811909 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.811886 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs"] Oct 31 00:56:59.847552 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.847518 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:56:59.849082 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.849066 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qw74k"] Oct 31 00:56:59.850970 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.850956 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.854374 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.854341 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:56:59.854485 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.854392 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:56:59.854546 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.854532 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:56:59.854602 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.854571 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ssrlm\"" Oct 31 00:56:59.858327 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.858307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:59.858327 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.858320 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:59.896779 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.896752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.896779 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.896784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.896988 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.896802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.896988 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.896819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.896988 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.896836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.897085 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.896999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-tls\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897085 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74t85\" (UniqueName: \"kubernetes.io/projected/fc981b83-c624-4524-95c0-950ec6a283ee-kube-api-access-74t85\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.897085 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-sys\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq5q4\" (UniqueName: \"kubernetes.io/projected/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-api-access-kq5q4\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.897230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-metrics-client-ca\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-root\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897230 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-wtmp\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897431 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897234 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.897431 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-textfile\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897431 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc981b83-c624-4524-95c0-950ec6a283ee-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.897431 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8cf\" (UniqueName: \"kubernetes.io/projected/a62692fc-3f77-478d-90d5-577b92632a39-kube-api-access-2x8cf\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897629 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.897629 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:56:59.897721 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.897721 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.897721 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.897888 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897751 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.898010 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.897987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.898188 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.898167 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc981b83-c624-4524-95c0-950ec6a283ee-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.899496 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.899471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.899907 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.899884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.900094 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.900075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.900160 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.900119 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.904091 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.904077 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:56:59.942362 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.942340 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hnbxm\"" Oct 31 00:56:59.949557 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.949510 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:56:59.950571 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.950555 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:56:59.951928 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.951905 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxv7v\"" Oct 31 00:56:59.953506 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.953477 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:56:59.953893 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.953855 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7j8s4\"" Oct 31 00:56:59.953999 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.953950 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:56:59.955662 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.955641 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq5q4\" (UniqueName: \"kubernetes.io/projected/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-api-access-kq5q4\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:56:59.955956 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.955937 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74t85\" (UniqueName: \"kubernetes.io/projected/fc981b83-c624-4524-95c0-950ec6a283ee-kube-api-access-74t85\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:56:59.960747 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.960724 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:56:59.998177 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-sys\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998177 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-metrics-client-ca\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-root\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-wtmp\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-textfile\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8cf\" (UniqueName: \"kubernetes.io/projected/a62692fc-3f77-478d-90d5-577b92632a39-kube-api-access-2x8cf\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.998414 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-tls\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.999684 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998830 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-wtmp\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.999684 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.998902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-sys\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.999684 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.999307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-textfile\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.999684 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.999375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-metrics-client-ca\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:56:59.999684 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:56:59.999424 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-root\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:00.000377 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.000306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:00.001253 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.001228 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-tls\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:00.002556 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.002524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:00.010667 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.010636 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:57:00.014327 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.014302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:00.027709 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.027667 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:00.055925 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.055845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8cf\" (UniqueName: \"kubernetes.io/projected/a62692fc-3f77-478d-90d5-577b92632a39-kube-api-access-2x8cf\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:00.060879 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.060726 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:00.163769 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.163746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:00.172729 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:00.172705 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62692fc_3f77_478d_90d5_577b92632a39.slice/crio-d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09 WatchSource:0}: Error finding container d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09: Status 404 returned error can't find the container with id d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09 Oct 31 00:57:00.247332 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.247266 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gjj6f\"" Oct 31 00:57:00.256309 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.256274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:00.261455 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.261429 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw74k" event={"ID":"a62692fc-3f77-478d-90d5-577b92632a39","Type":"ContainerStarted","Data":"d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09"} Oct 31 00:57:00.326057 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.325964 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-48ddr"] Oct 31 00:57:00.328179 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:00.328147 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bb82ee_c3c0_4719_8a69_04ac4ce085cd.slice/crio-a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0 WatchSource:0}: Error finding container a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0: Status 404 returned error can't find the container with id a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0 Oct 31 00:57:00.374844 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.374817 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pgqj5"] Oct 31 00:57:00.377096 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:00.377072 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fda59f9_289b_431d_ba24_c291169fe1ae.slice/crio-ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764 WatchSource:0}: Error finding container ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764: Status 404 returned error can't find the container with id ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764 Oct 31 00:57:00.429427 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.429400 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7"] Oct 31 00:57:00.432703 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:00.432676 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc981b83_c624_4524_95c0_950ec6a283ee.slice/crio-6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3 WatchSource:0}: Error finding container 6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3: Status 404 returned error can't find the container with id 6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3 Oct 31 00:57:00.438734 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.438688 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs"] Oct 31 00:57:00.441362 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:00.441339 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7da2d7d4_64fd_4ecf_a842_70e58f3a6bf2.slice/crio-73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583 WatchSource:0}: Error finding container 73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583: Status 404 returned error can't find the container with id 73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583 Oct 31 00:57:00.625070 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:00.625043 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-blwrh"] Oct 31 00:57:00.627228 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:00.627202 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8189bdc_c813_4a8d_a1e0_052a4415d4ba.slice/crio-e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f WatchSource:0}: Error finding container e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f: Status 404 returned error can't find the container with id e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f Oct 31 00:57:01.268976 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.268927 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" event={"ID":"fc981b83-c624-4524-95c0-950ec6a283ee","Type":"ContainerStarted","Data":"a5602e64b815eecbceb36318a80b53814204efb3c2184d420b82b3ae867d5ee0"} Oct 31 00:57:01.269423 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.269006 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" event={"ID":"fc981b83-c624-4524-95c0-950ec6a283ee","Type":"ContainerStarted","Data":"54fdb0405049e8528b33acbc652ce7e89f1d89402283bfe965e6dc2df950c08f"} Oct 31 00:57:01.269423 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.269059 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" event={"ID":"fc981b83-c624-4524-95c0-950ec6a283ee","Type":"ContainerStarted","Data":"6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3"} Oct 31 00:57:01.270389 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.270363 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-48ddr" event={"ID":"66bb82ee-c3c0-4719-8a69-04ac4ce085cd","Type":"ContainerStarted","Data":"a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0"} Oct 31 00:57:01.272889 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.272846 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" event={"ID":"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2","Type":"ContainerStarted","Data":"73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583"} Oct 31 00:57:01.274243 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.274167 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-blwrh" event={"ID":"d8189bdc-c813-4a8d-a1e0-052a4415d4ba","Type":"ContainerStarted","Data":"e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f"} Oct 31 00:57:01.275490 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:01.275470 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgqj5" event={"ID":"0fda59f9-289b-431d-ba24-c291169fe1ae","Type":"ContainerStarted","Data":"ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764"} Oct 31 00:57:02.132171 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.132081 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:02.135120 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.135098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215389 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215532 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215750 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215750 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215750 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.215750 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.215616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.285844 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.285805 2578 generic.go:356] "Generic (PLEG): container finished" podID="a62692fc-3f77-478d-90d5-577b92632a39" containerID="846b3a840bb9e1237c381a4f1a223aff4197a3c0ee8e1dd2688757d781fd4ca5" exitCode=0 Oct 31 00:57:02.286589 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.286563 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw74k" event={"ID":"a62692fc-3f77-478d-90d5-577b92632a39","Type":"ContainerDied","Data":"846b3a840bb9e1237c381a4f1a223aff4197a3c0ee8e1dd2688757d781fd4ca5"} Oct 31 00:57:02.293605 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.293577 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:57:02.293721 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.293673 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:57:02.293789 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.293750 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:57:02.295758 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.294879 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:57:02.295758 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.294957 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:57:02.295758 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.295012 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:57:02.295758 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.295063 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:57:02.295758 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.295139 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:57:02.295758 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.295230 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:57:02.296359 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.293857 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kc974\"" Oct 31 00:57:02.298192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.298075 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.317115 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.316648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.319256 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.319223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.319660 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.319642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.320398 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.320357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.323061 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.323039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.324699 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.324647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.325161 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.325143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.325410 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.325379 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.343030 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.343003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.343296 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.343273 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.343691 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.343636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.343786 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.343681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.347736 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.347685 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.377336 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.377309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.457319 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.455607 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69f8cc8d77-825m2"] Oct 31 00:57:02.457319 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.457182 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:02.460274 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.460108 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.521893 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.521046 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:57:02.521893 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.521100 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7o3ddrn43kmg7\"" Oct 31 00:57:02.521893 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.521162 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:57:02.521893 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.521264 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:57:02.521893 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.521613 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rsnwq\"" Oct 31 00:57:02.522210 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.522074 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:57:02.524286 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.524252 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.527810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528197 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c49c201-987e-4ea9-9bec-343a09101685-metrics-client-ca\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c49c201-987e-4ea9-9bec-343a09101685-tmp\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-grpc-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.528886 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pwc\" (UniqueName: \"kubernetes.io/projected/2c49c201-987e-4ea9-9bec-343a09101685-kube-api-access-k8pwc\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.534192 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.529031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.537790 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.537759 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69f8cc8d77-825m2"] Oct 31 00:57:02.630538 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630500 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-grpc-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630689 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630689 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630689 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pwc\" (UniqueName: \"kubernetes.io/projected/2c49c201-987e-4ea9-9bec-343a09101685-kube-api-access-k8pwc\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630689 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630807 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630807 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630885 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c49c201-987e-4ea9-9bec-343a09101685-metrics-client-ca\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.630885 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.630837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c49c201-987e-4ea9-9bec-343a09101685-tmp\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.631270 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.631249 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c49c201-987e-4ea9-9bec-343a09101685-tmp\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.631973 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.631881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c49c201-987e-4ea9-9bec-343a09101685-metrics-client-ca\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.641486 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.641426 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.641486 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.641466 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.642070 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.642044 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.642147 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.642072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.642216 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.642171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-grpc-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.645514 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.645493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.693476 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.693423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pwc\" (UniqueName: \"kubernetes.io/projected/2c49c201-987e-4ea9-9bec-343a09101685-kube-api-access-k8pwc\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.846947 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.846909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:02.934440 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:02.929202 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:02.934440 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:02.933355 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6bf6ec7_e5e0_4adf_ae7b_7d7e707ee66d.slice/crio-551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8 WatchSource:0}: Error finding container 551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8: Status 404 returned error can't find the container with id 551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8 Oct 31 00:57:03.236799 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.236710 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:03.241178 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.241126 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69f8cc8d77-825m2"] Oct 31 00:57:03.243290 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:03.243263 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c49c201_987e_4ea9_9bec_343a09101685.slice/crio-6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed WatchSource:0}: Error finding container 6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed: Status 404 returned error can't find the container with id 6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed Oct 31 00:57:03.292678 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.292636 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed"} Oct 31 00:57:03.295484 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.295457 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" event={"ID":"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2","Type":"ContainerStarted","Data":"01f54be2334516c29caefc4e910bf3f940696737b643afcf4bfc7383cea508f7"} Oct 31 00:57:03.295619 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.295517 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" event={"ID":"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2","Type":"ContainerStarted","Data":"8e5b3d715428673275cd70dbe2223b501f0f8c4e044344c353da8d6347a3014c"} Oct 31 00:57:03.295619 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.295544 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" event={"ID":"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2","Type":"ContainerStarted","Data":"87d4fdace16587c6e139c8a53e1711a47391808703bc7fa3ab185e1c6d3006ca"} Oct 31 00:57:03.297488 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.297466 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerStarted","Data":"551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8"} Oct 31 00:57:03.300457 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.300431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgqj5" event={"ID":"0fda59f9-289b-431d-ba24-c291169fe1ae","Type":"ContainerStarted","Data":"9df198961dbc996ace4626c6998e6125bf9366038155901081dcabf61faaded3"} Oct 31 00:57:03.300563 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.300485 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pgqj5" event={"ID":"0fda59f9-289b-431d-ba24-c291169fe1ae","Type":"ContainerStarted","Data":"91e12531bd69c4b89b717567bad2a83e756d009547b00f4055567480e27a4d35"} Oct 31 00:57:03.303236 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.303211 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw74k" event={"ID":"a62692fc-3f77-478d-90d5-577b92632a39","Type":"ContainerStarted","Data":"b755565d231e315c2c53b84c28233faa46fa969caa1d3de5b89f4f7700783d8b"} Oct 31 00:57:03.303324 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.303255 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qw74k" event={"ID":"a62692fc-3f77-478d-90d5-577b92632a39","Type":"ContainerStarted","Data":"bb57a9ff84c9a1e7b305168f98709e46f6ece29f8653eec3350c70ae755befc7"} Oct 31 00:57:03.306547 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.306526 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" event={"ID":"fc981b83-c624-4524-95c0-950ec6a283ee","Type":"ContainerStarted","Data":"b336224eda2c8f843dd9f20283caf93011420d752c2685bc5d09460bd135f609"} Oct 31 00:57:03.480669 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.480602 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" podStartSLOduration=2.05677914 podStartE2EDuration="4.480582577s" totalImagesPullingTime="2.423803437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:03.480159233 +0000 UTC m=+70.028965859" watchObservedRunningTime="2026-10-31 00:57:03.480582577 +0000 UTC m=+70.029389213" Oct 31 00:57:03.723639 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.722714 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qw74k" podStartSLOduration=3.015137164 podStartE2EDuration="4.722693743s" totalImagesPullingTime="1.615664316s" totalInitContainerRuntime="91.892263ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:03.603893011 +0000 UTC m=+70.152699646" watchObservedRunningTime="2026-10-31 00:57:03.722693743 +0000 UTC m=+70.271500379" Oct 31 00:57:03.723639 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.722909 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pgqj5" podStartSLOduration=67.746338464 podStartE2EDuration="1m9.722897035s" totalImagesPullingTime="1.976558571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:03.721532879 +0000 UTC m=+70.270339516" watchObservedRunningTime="2026-10-31 00:57:03.722897035 +0000 UTC m=+70.271703668" Oct 31 00:57:03.843338 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:03.841990 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" podStartSLOduration=3.005503119 podStartE2EDuration="4.841970311s" totalImagesPullingTime="1.836467192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:03.841162235 +0000 UTC m=+70.389968870" watchObservedRunningTime="2026-10-31 00:57:03.841970311 +0000 UTC m=+70.390776947" Oct 31 00:57:04.519980 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.519942 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk"] Oct 31 00:57:04.522477 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.522452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.550381 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.550224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7mb\" (UniqueName: \"kubernetes.io/projected/73310d59-0bee-4124-ad47-b480d3544468-kube-api-access-jf7mb\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.550381 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.550289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73310d59-0bee-4124-ad47-b480d3544468-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.586274 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.586215 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk"] Oct 31 00:57:04.586770 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.586651 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:57:04.586770 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.586738 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rs8xl\"" Oct 31 00:57:04.650836 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.650810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7mb\" (UniqueName: \"kubernetes.io/projected/73310d59-0bee-4124-ad47-b480d3544468-kube-api-access-jf7mb\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.651006 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.650854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73310d59-0bee-4124-ad47-b480d3544468-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.655649 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.655597 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73310d59-0bee-4124-ad47-b480d3544468-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.677341 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.677315 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cf56d4544-tzfbs"] Oct 31 00:57:04.679466 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.679447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.710185 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.710130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7mb\" (UniqueName: \"kubernetes.io/projected/73310d59-0bee-4124-ad47-b480d3544468-kube-api-access-jf7mb\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.734451 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.733827 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:57:04.734451 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.733968 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:57:04.734451 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.734202 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6bhn88i2ditsv\"" Oct 31 00:57:04.734451 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.734289 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:57:04.734451 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.734363 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-n4snk\"" Oct 31 00:57:04.734451 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.734403 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:57:04.743067 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.742946 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cf56d4544-tzfbs"] Oct 31 00:57:04.752269 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-metrics-server-audit-profiles\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.752269 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7m2\" (UniqueName: \"kubernetes.io/projected/236b6fa3-674c-4790-951e-8b91cb51deee-kube-api-access-xf7m2\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.752458 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-client-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.752458 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752393 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-client-certs\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.752458 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/236b6fa3-674c-4790-951e-8b91cb51deee-audit-log\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.752601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.752601 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.752488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-tls\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.836923 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.836890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:04.854107 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-metrics-server-audit-profiles\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.854222 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7m2\" (UniqueName: \"kubernetes.io/projected/236b6fa3-674c-4790-951e-8b91cb51deee-kube-api-access-xf7m2\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.854222 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-client-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.854336 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-client-certs\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.854336 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/236b6fa3-674c-4790-951e-8b91cb51deee-audit-log\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.854336 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854281 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.854336 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.854308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-tls\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.855944 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.855194 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/236b6fa3-674c-4790-951e-8b91cb51deee-audit-log\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.855944 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.855856 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.856543 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.856503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-metrics-server-audit-profiles\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.857578 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.857525 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-tls\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.858671 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.858651 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-client-certs\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.860125 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.860069 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-client-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:04.916664 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:04.916638 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7m2\" (UniqueName: \"kubernetes.io/projected/236b6fa3-674c-4790-951e-8b91cb51deee-kube-api-access-xf7m2\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:05.017406 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.017372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:05.145148 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.143633 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-698867449c-xbf2l"] Oct 31 00:57:05.146389 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.146359 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.252937 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.252902 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:57:05.254389 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.254346 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698867449c-xbf2l"] Oct 31 00:57:05.255615 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.255595 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk"] Oct 31 00:57:05.257009 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:05.256983 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73310d59_0bee_4124_ad47_b480d3544468.slice/crio-ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125 WatchSource:0}: Error finding container ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125: Status 404 returned error can't find the container with id ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125 Oct 31 00:57:05.257195 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257172 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257285 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257285 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257399 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257374 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257492 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257578 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257578 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.257666 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.257598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.319606 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.319568 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" exitCode=0 Oct 31 00:57:05.319770 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.319711 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} Oct 31 00:57:05.322045 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.322022 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" event={"ID":"73310d59-0bee-4124-ad47-b480d3544468","Type":"ContainerStarted","Data":"ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125"} Oct 31 00:57:05.358876 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.358818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359026 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.358943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359026 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.358989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359026 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359259 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359259 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359259 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.359428 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.360808 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.360808 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.359890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.360808 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.360443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.360808 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.360565 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.361768 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.361723 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.362526 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.362505 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.362833 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.362754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.403848 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.403697 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cf56d4544-tzfbs"] Oct 31 00:57:05.407470 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:05.407438 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236b6fa3_674c_4790_951e_8b91cb51deee.slice/crio-b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890 WatchSource:0}: Error finding container b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890: Status 404 returned error can't find the container with id b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890 Oct 31 00:57:05.419933 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.419847 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.459376 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.459128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:05.486200 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.485410 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:05.486200 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.485738 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:05.491605 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.491410 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:05.848006 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:05.847985 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-698867449c-xbf2l"] Oct 31 00:57:05.851031 ip-10-0-136-249 kubenswrapper[2578]: W1031 00:57:05.851000 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d183d8_26d7_4e77_aa32_64e75fc435a5.slice/crio-01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432 WatchSource:0}: Error finding container 01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432: Status 404 returned error can't find the container with id 01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432 Oct 31 00:57:06.330446 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.330410 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-blwrh" event={"ID":"d8189bdc-c813-4a8d-a1e0-052a4415d4ba","Type":"ContainerStarted","Data":"f63c6e950cc23c301e37eaae21cf1943994b0ab8de48634cc1472ce97b1e4dbe"} Oct 31 00:57:06.330984 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.330963 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:06.334785 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.334721 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698867449c-xbf2l" event={"ID":"f3d183d8-26d7-4e77-aa32-64e75fc435a5","Type":"ContainerStarted","Data":"439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256"} Oct 31 00:57:06.334785 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.334763 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698867449c-xbf2l" event={"ID":"f3d183d8-26d7-4e77-aa32-64e75fc435a5","Type":"ContainerStarted","Data":"01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432"} Oct 31 00:57:06.337111 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.337084 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" event={"ID":"236b6fa3-674c-4790-951e-8b91cb51deee","Type":"ContainerStarted","Data":"b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890"} Oct 31 00:57:06.342326 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.342302 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:06.449040 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.448542 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-blwrh" podStartSLOduration=67.502625268 podStartE2EDuration="1m12.448520825s" totalImagesPullingTime="4.945895557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:06.447071889 +0000 UTC m=+72.995878525" watchObservedRunningTime="2026-10-31 00:57:06.448520825 +0000 UTC m=+72.997327460" Oct 31 00:57:06.704857 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:06.704783 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-698867449c-xbf2l" podStartSLOduration=1.704763142 podStartE2EDuration="1.704763142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:06.704189963 +0000 UTC m=+73.252996602" watchObservedRunningTime="2026-10-31 00:57:06.704763142 +0000 UTC m=+73.253569777" Oct 31 00:57:07.346501 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:07.346461 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"f31e06cec001251dc2d7391f59fcdce19826aaa3e84a14b3b3c039af1c61fd9a"} Oct 31 00:57:07.346999 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:07.346548 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"772985d2ca2d5e0c29cd5713768564a95d979f6f7095c50b6ba64a00d4a123df"} Oct 31 00:57:07.346999 ip-10-0-136-249 kubenswrapper[2578]: I1031 00:57:07.346592 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"ebea8291a6ace1591c2ad5228bb35030311ea9ed4be3026099c5812a8e6ceb88"} Oct 31 00:57:08.116890 ip-10-0-136-249 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 00:57:08.136315 ip-10-0-136-249 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 00:57:08.136523 ip-10-0-136-249 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 00:57:08.136576 ip-10-0-136-249 systemd[1]: kubelet.service: Consumed 4.200s CPU time. Oct 31 00:57:08.155663 ip-10-0-136-249 systemd[1]: Starting Kubernetes Kubelet... Oct 31 00:57:08.222675 ip-10-0-136-249 kubenswrapper[5756]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:57:08.222675 ip-10-0-136-249 kubenswrapper[5756]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 00:57:08.222675 ip-10-0-136-249 kubenswrapper[5756]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:57:08.222675 ip-10-0-136-249 kubenswrapper[5756]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 00:57:08.225845 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225756 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.225845 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225843 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.225845 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225848 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.225845 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225851 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.225845 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225854 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225857 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225875 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225878 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225880 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225883 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225886 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225889 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225891 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225894 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225898 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225901 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225903 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225906 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225909 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225911 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225914 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225917 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225919 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225922 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.226041 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225925 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225927 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225930 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225932 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225934 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225937 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225939 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225946 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225949 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225951 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225954 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225957 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225959 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225962 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225964 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225967 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225970 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225973 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225977 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.226500 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225981 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225983 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225986 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225989 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225991 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225994 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.225996 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226000 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226002 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226006 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226008 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226011 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226013 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226016 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226018 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226020 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226024 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226029 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226031 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.226951 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226034 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226036 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226039 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226043 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226046 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226049 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226052 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226054 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226057 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226060 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226062 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226064 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226067 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226070 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226072 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226075 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226077 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226079 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226082 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.227513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226084 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226086 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226090 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226093 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226095 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226099 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226101 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226105 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226107 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226110 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226112 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226115 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226117 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226120 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226123 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226126 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226129 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226131 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226134 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226136 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.228289 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226139 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226141 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226144 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226146 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226148 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226898 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226905 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226908 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226911 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226914 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226916 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226919 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226922 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226924 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226927 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226931 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226935 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226938 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226940 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226943 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.229077 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226946 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226948 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226951 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226954 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226956 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226959 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226961 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226964 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226966 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226969 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226971 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226974 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226977 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226979 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226982 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226984 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226987 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226989 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226992 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226994 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.229937 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226997 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.226999 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227001 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227004 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227007 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227009 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227012 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227015 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227018 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227021 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227023 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227025 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227029 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227033 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227036 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227039 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227042 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227045 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227048 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.230748 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227050 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227053 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227055 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227058 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227061 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227065 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227069 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227073 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227077 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227081 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227085 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227089 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227093 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227097 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227103 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227108 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227113 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227118 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227122 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.231562 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227127 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227132 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227136 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227141 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227145 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227149 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227153 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227157 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227161 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227165 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227170 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227174 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227178 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227182 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227186 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227190 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227195 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227199 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227203 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.232391 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227207 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227211 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227215 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227219 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227223 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227227 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227231 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227236 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227240 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227247 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227251 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227255 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227263 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.227267 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228471 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228481 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228484 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228487 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228490 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228493 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.233183 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228496 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228499 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228501 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228504 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228507 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228509 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228512 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228516 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228520 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228525 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228529 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228533 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228536 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228540 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228545 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228549 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228553 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228557 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228562 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228565 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.234083 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228568 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228571 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228573 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228575 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228580 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228583 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228586 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228589 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228591 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228594 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228597 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228600 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228605 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228609 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228612 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228614 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228617 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228619 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228622 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.234935 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228624 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228627 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228630 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228633 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228636 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228639 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228642 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228644 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228647 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228650 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228653 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228655 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228658 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228660 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228663 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228666 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228668 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228672 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228675 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228678 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.235707 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228680 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228683 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228686 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228691 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228695 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228700 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228704 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228707 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228709 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228712 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228714 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228717 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228719 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228722 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228724 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228727 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228730 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228732 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228735 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.236417 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228737 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228740 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228743 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228746 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228750 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228755 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228759 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228763 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228768 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228771 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228777 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228782 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228786 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228790 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228794 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228798 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228800 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228804 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228807 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.237249 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228810 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228812 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.228815 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228948 5756 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228963 5756 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228970 5756 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228974 5756 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228979 5756 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228982 5756 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228986 5756 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228990 5756 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228993 5756 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.228996 5756 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229000 5756 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229004 5756 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229007 5756 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229010 5756 flags.go:64] FLAG: --cgroup-root="" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229013 5756 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229016 5756 flags.go:64] FLAG: --client-ca-file="" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229019 5756 flags.go:64] FLAG: --cloud-provider="external" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229022 5756 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229026 5756 flags.go:64] FLAG: --cluster-domain="" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229029 5756 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 00:57:08.237827 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229034 5756 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229037 5756 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229040 5756 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229048 5756 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229051 5756 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229055 5756 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229058 5756 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229061 5756 flags.go:64] FLAG: --contention-profiling="false" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229064 5756 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229067 5756 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229070 5756 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229073 5756 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229077 5756 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229081 5756 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229084 5756 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229087 5756 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229090 5756 flags.go:64] FLAG: --enable-server="true" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229093 5756 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229099 5756 flags.go:64] FLAG: --event-burst="100" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229102 5756 flags.go:64] FLAG: --event-qps="50" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229105 5756 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229108 5756 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229111 5756 flags.go:64] FLAG: --eviction-hard="" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229115 5756 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 00:57:08.238480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229118 5756 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229121 5756 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229125 5756 flags.go:64] FLAG: --eviction-soft="" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229127 5756 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229130 5756 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229133 5756 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229136 5756 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229139 5756 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229142 5756 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229145 5756 flags.go:64] FLAG: --feature-gates="" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229149 5756 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229152 5756 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229155 5756 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229158 5756 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229161 5756 flags.go:64] FLAG: --healthz-port="10248" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229164 5756 flags.go:64] FLAG: --help="false" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229167 5756 flags.go:64] FLAG: --hostname-override="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229171 5756 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229174 5756 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229177 5756 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229180 5756 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229183 5756 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229186 5756 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 00:57:08.239296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229189 5756 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229191 5756 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229194 5756 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229197 5756 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229200 5756 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229203 5756 flags.go:64] FLAG: --kube-reserved="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229206 5756 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229208 5756 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229211 5756 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229214 5756 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229216 5756 flags.go:64] FLAG: --lock-file="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229219 5756 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229222 5756 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229225 5756 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229230 5756 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229233 5756 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229236 5756 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229239 5756 flags.go:64] FLAG: --logging-format="text" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229242 5756 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229245 5756 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229248 5756 flags.go:64] FLAG: --manifest-url="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229251 5756 flags.go:64] FLAG: --manifest-url-header="" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229259 5756 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229262 5756 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229266 5756 flags.go:64] FLAG: --max-pods="110" Oct 31 00:57:08.240292 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229269 5756 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229272 5756 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229276 5756 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229278 5756 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229281 5756 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229284 5756 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229287 5756 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229295 5756 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229298 5756 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229301 5756 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229304 5756 flags.go:64] FLAG: --pod-cidr="" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229307 5756 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229310 5756 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229313 5756 flags.go:64] FLAG: --pods-per-core="0" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229316 5756 flags.go:64] FLAG: --port="10250" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229319 5756 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229322 5756 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-07c8891944f7ab7ce" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229325 5756 flags.go:64] FLAG: --qos-reserved="" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229328 5756 flags.go:64] FLAG: --read-only-port="10255" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229330 5756 flags.go:64] FLAG: --register-node="true" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229334 5756 flags.go:64] FLAG: --register-with-taints="" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229337 5756 flags.go:64] FLAG: --registry-burst="10" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229340 5756 flags.go:64] FLAG: --registry-qps="5" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229343 5756 flags.go:64] FLAG: --reserved-cpus="" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229346 5756 flags.go:64] FLAG: --reserved-memory="" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229350 5756 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 00:57:08.241648 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229353 5756 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229356 5756 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229359 5756 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229362 5756 flags.go:64] FLAG: --runonce="false" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229365 5756 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229368 5756 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229371 5756 flags.go:64] FLAG: --seccomp-default="false" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229373 5756 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229376 5756 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229380 5756 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229383 5756 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229386 5756 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229389 5756 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229393 5756 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229396 5756 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229399 5756 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229402 5756 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229405 5756 flags.go:64] FLAG: --system-cgroups="" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229408 5756 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229413 5756 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229415 5756 flags.go:64] FLAG: --tls-cert-file="" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229418 5756 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229422 5756 flags.go:64] FLAG: --tls-min-version="" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229425 5756 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229427 5756 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 00:57:08.242426 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229430 5756 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229433 5756 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229436 5756 flags.go:64] FLAG: --v="2" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229440 5756 flags.go:64] FLAG: --version="false" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229444 5756 flags.go:64] FLAG: --vmodule="" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229448 5756 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229451 5756 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229593 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229597 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229600 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229604 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229607 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229610 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229612 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229614 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229617 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229619 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229622 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229625 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229629 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229633 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.243196 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229636 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229638 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229641 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229644 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229646 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229649 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229651 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229654 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229657 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229660 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229662 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229664 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229667 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229669 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229672 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229674 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229677 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229679 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229681 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.243680 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229684 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229686 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229688 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229691 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229693 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229696 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229698 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229700 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229703 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229705 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229708 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229713 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229716 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229719 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229721 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229724 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229726 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229729 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229731 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229734 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.244168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229736 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229739 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229741 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229744 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229746 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229748 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229751 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229753 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229756 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229758 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229761 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229763 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229765 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229768 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229771 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229773 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229776 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229778 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229781 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229783 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.244766 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229786 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229788 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229790 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229801 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229805 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229808 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229810 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229813 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229815 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229817 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229820 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229822 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229825 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229828 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229832 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229835 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229838 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229841 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229843 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.245351 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229846 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229848 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229851 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229853 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229856 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229874 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229878 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229881 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229884 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229886 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229889 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229892 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229895 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229897 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.229924 5756 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229960 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.245807 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229965 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229973 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229976 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229979 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229981 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229984 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229986 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229989 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229991 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229994 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229997 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.229999 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230002 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230005 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230007 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230010 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230012 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230015 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230017 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230020 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.246200 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230022 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230025 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230027 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230029 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230032 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230034 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230037 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230039 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230042 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230044 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230046 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230049 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230052 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230056 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230059 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230062 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230065 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230067 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230069 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230072 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.246660 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230074 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230077 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230080 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230082 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230084 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230087 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230089 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230092 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230094 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230097 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230099 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230102 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230104 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230106 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230109 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230111 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230115 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230119 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230122 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.247177 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230125 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230128 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230130 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230133 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230135 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230137 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230142 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230145 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230149 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230152 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230155 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230157 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230160 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230163 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230165 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230168 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230170 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230173 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230175 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230177 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.247653 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230180 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230182 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230185 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230187 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230190 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230192 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230195 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230197 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230200 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230202 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230205 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230207 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230210 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230213 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230215 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230218 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230220 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230223 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230228 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.248168 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230230 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230234 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230237 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230240 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230242 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230244 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.230247 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.234252 5756 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.234268 5756 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.234285 5756 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.234290 5756 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234348 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234353 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234357 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234360 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234362 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234365 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234368 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234370 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234373 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.248611 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234375 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234378 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234380 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234383 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234385 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234388 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234392 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234394 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234397 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234400 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234403 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234405 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234408 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234410 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234413 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234415 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234418 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234421 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234423 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.249101 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234425 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234429 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234432 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234434 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234437 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234439 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234442 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234444 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234447 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234449 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234451 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234454 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234458 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234462 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234465 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234468 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234471 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234474 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234476 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234479 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.249548 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234481 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234484 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234486 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234489 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234491 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234493 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234496 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234499 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234501 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234504 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234506 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234509 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234512 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234514 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234518 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234520 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234523 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234525 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234528 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.250024 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234530 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234533 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234535 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234538 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234540 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234543 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234545 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234548 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234550 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234553 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234555 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234559 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234563 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234566 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234568 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234571 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234573 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234578 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234581 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.250469 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234584 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234586 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234589 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234591 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234594 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234596 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234599 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234602 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234604 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234608 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234610 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234613 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234616 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234618 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234621 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234623 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234626 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234628 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234631 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.251034 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234633 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.234660 5756 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234834 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234839 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234842 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234845 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234847 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234850 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234853 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234856 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234875 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234879 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234883 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234886 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234889 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234892 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.251698 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234894 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234897 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234899 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234902 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234904 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234907 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234909 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234919 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234922 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234924 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234927 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234929 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234932 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234935 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234937 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234940 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234942 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234945 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234948 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234950 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.252284 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234953 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234956 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234958 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234961 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234964 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234966 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234968 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234971 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234974 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234977 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234979 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234982 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234984 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234986 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234989 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234991 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234994 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234996 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.234999 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.252913 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235001 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235009 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235011 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235014 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235016 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235019 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235021 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235023 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235026 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235028 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235031 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235033 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235036 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235038 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235041 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235043 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235046 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235048 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235050 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235053 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.253677 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235055 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235058 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235067 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235069 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235072 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235074 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235077 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235079 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235082 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235084 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235087 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235089 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235091 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235094 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235102 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235105 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235107 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235110 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235112 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235116 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.254415 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235119 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235122 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235124 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235129 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235131 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235134 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235138 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235141 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235144 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235146 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235149 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235151 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235153 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.235177 5756 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235216 5756 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 00:57:08.255110 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235219 5756 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235222 5756 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235225 5756 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235228 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235230 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235233 5756 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235235 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235238 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235240 5756 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235242 5756 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235245 5756 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235248 5756 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235256 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235259 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235261 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235264 5756 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235266 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235269 5756 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235271 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235274 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 00:57:08.255511 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235278 5756 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235280 5756 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235283 5756 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235286 5756 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235288 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235291 5756 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235293 5756 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235295 5756 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235298 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235300 5756 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235303 5756 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235305 5756 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235308 5756 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235310 5756 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235313 5756 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235315 5756 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235317 5756 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235320 5756 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235323 5756 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235325 5756 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 00:57:08.256004 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235327 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235330 5756 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235333 5756 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235337 5756 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235340 5756 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235348 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235352 5756 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235355 5756 feature_gate.go:386] unrecognized feature gate: Example Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235357 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235362 5756 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235365 5756 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235368 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235372 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235375 5756 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235377 5756 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235380 5756 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235382 5756 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235385 5756 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235387 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 00:57:08.256513 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235390 5756 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235392 5756 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235397 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235399 5756 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235402 5756 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235404 5756 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235407 5756 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235409 5756 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235412 5756 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235415 5756 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235417 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235420 5756 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235422 5756 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235425 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235427 5756 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235430 5756 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235433 5756 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235435 5756 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235437 5756 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 00:57:08.257144 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235447 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235449 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235452 5756 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235454 5756 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235457 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235459 5756 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235463 5756 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235465 5756 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235468 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235471 5756 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235474 5756 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235476 5756 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235478 5756 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235481 5756 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235483 5756 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235487 5756 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235489 5756 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235492 5756 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235494 5756 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 00:57:08.257874 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235497 5756 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235499 5756 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235501 5756 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235504 5756 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235506 5756 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235509 5756 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235511 5756 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:08.235514 5756 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.235587 5756 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.236603 5756 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.236659 5756 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.236909 5756 server.go:1014] "Starting client certificate rotation" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.237131 5756 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.237257 5756 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:50:54 +0000 UTC" deadline="2028-04-30 01:03:06.066294217 +0000 UTC" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.237313 5756 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13128h5m57.828983337s" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.237689 5756 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:57:08.258626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.240616 5756 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 00:57:08.259205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.242345 5756 log.go:25] "Validated CRI v1 runtime API" Oct 31 00:57:08.259205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.244305 5756 log.go:25] "Validated CRI v1 image API" Oct 31 00:57:08.259205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.245067 5756 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 00:57:08.259205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.247228 5756 fs.go:135] Filesystem UUIDs: map[2eec57ee-b68f-4c32-b47f-34294ad357f9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f61103ec-55f2-4caa-bfa2-ec694fe8eb8a:/dev/nvme0n1p3] Oct 31 00:57:08.259400 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.247247 5756 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed/userdata/shm major:0 minor:369 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125/userdata/shm major:0 minor:396 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01262e31-9355-45a2-adc6-2f7889c4fe8e/volumes/kubernetes.io~projected/kube-api-access-2fcnq:{mountpoint:/var/lib/kubelet/pods/01262e31-9355-45a2-adc6-2f7889c4fe8e/volumes/kubernetes.io~projected/kube-api-access-2fcnq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01262e31-9355-45a2-adc6-2f7889c4fe8e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/01262e31-9355-45a2-adc6-2f7889c4fe8e/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fda59f9-289b-431d-ba24-c291169fe1ae/volumes/kubernetes.io~projected/kube-api-access-gdhvt:{mountpoint:/var/lib/kubelet/pods/0fda59f9-289b-431d-ba24-c291169fe1ae/volumes/kubernetes.io~projected/kube-api-access-gdhvt major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fda59f9-289b-431d-ba24-c291169fe1ae/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0fda59f9-289b-431d-ba24-c291169fe1ae/volumes/kubernetes.io~secret/metrics-certs major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10f64c92-95d2-4fc4-add2-d174ec7fe24f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/10f64c92-95d2-4fc4-add2-d174ec7fe24f/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1f47fdf3-f591-4184-abff-d4670f404937/volumes/kubernetes.io~projected/kube-api-access-w2lts:{mountpoint:/var/lib/kubelet/pods/1f47fdf3-f591-4184-abff-d4670f404937/volumes/kubernetes.io~projected/kube-api-access-w2lts major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~projected/kube-api-access-xf7m2:{mountpoint:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~projected/kube-api-access-xf7m2 major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~projected/kube-api-access-k8pwc:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~projected/kube-api-access-k8pwc major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3537b0f9-a07f-423d-a9e2-8a3abc3c3916/volumes/kubernetes.io~projected/kube-api-access-4d2k2:{mountpoint:/var/lib/kubelet/pods/3537b0f9-a07f-423d-a9e2-8a3abc3c3916/volumes/kubernetes.io~projected/kube-api-access-4d2k2 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~projected/kube-api-access-pphfn:{mountpoint:/var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~projected/kube-api-access-pphfn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41b41813-6309-49be-bd2c-76b0820899af/volumes/kubernetes.io~projected/kube-api-access-n5spv:{mountpoint:/var/lib/kubelet/pods/41b41813-6309-49be-bd2c-76b0820899af/volumes/kubernetes.io~projected/kube-api-access-n5spv major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41b41813-6309-49be-bd2c-76b0820899af/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/41b41813-6309-49be-bd2c-76b0820899af/volumes/kubernetes.io~secret/metrics-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volumes/kubernetes.io~projected/kube-api-access-vn28c:{mountpoint:/var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volumes/kubernetes.io~projected/kube-api-access-vn28c major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63419a67-aaca-408d-bcfa-7c302b749c9d/volumes/kubernetes.io~projected/kube-api-access-nzsz8:{mountpoint:/var/lib/kubelet/pods/63419a67-aaca-408d-bcfa-7c302b749c9d/volumes/kubernetes.io~projected/kube-api-access-nzsz8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~projected/kube-api-access-h2b8r:{mountpoint:/var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~projected/kube-api-access-h2b8r major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66bb82ee-c3c0-4719-8a69-04ac4ce085cd/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/66bb82ee-c3c0-4719-8a69-04ac4ce085cd/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73310d59-0bee-4124-ad47-b480d3544468/volumes/kubernetes.io~projected/kube-api-access-jf7mb:{mountpoint:/var/lib/kubelet/pods/73310d59-0bee-4124-ad47-b480d3544468/volumes/kubernetes.io~projected/kube-api-access-jf7mb major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73310d59-0bee-4124-ad47-b480d3544468/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/73310d59-0bee-4124-ad47-b480d3544468/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~projected/kube-api-access-kq5q4:{mountpoint:/var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~projected/kube-api-access-kq5q4 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~projected/kube-api-access-2x8cf:{mountpoint:/var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~projected/kube-api-access-2x8cf major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~projected/kube-api-access-xkvz8:{mountpoint:/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~projected/kube-api-access-xkvz8 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/alertmanager/9 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/config-reloader/6 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~projected/kube-api-access-2n7pp:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~projected/kube-api-access-2n7pp major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~projected/tls-assets major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/config-volume major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/web-config major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbaba560-5774-4a19-afde-447fb6548557/volumes/kubernetes.io~projected/kube-api-access-hl4lx:{mountpoint:/var/lib/kubelet/pods/cbaba560-5774-4a19-afde-447fb6548557/volumes/kubernetes.io~projected/kube-api-access-hl4lx major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbaba560-5774-4a19-afde-447fb6548557/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/cbaba560-5774-4a19-afde-447fb6548557/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25f0afe-e3c3-4ded-ab76-c98083607e36/volumes/kubernetes.io~projected/kube-api-access-qnx6j:{mountpoint:/var/lib/kubelet/pods/d25f0afe-e3c3-4ded-ab76-c98083607e36/volumes/kubernetes.io~projected/kube-api-access-qnx6j major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8189bdc-c813-4a8d-a1e0-052a4415d4ba/volumes/kubernetes.io~projected/kube-api-access-58psx:{mountpoint:/var/lib/kubelet/pods/d8189bdc-c813-4a8d-a1e0-052a4415d4ba/volumes/kubernetes.io~projected/kube-api-access-58psx major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e951e6b3-30a9-4181-912e-1a40f1725862/volumes/kubernetes.io~projected/kube-api-access-577tv:{mountpoint:/var/lib/kubelet/pods/e951e6b3-30a9-4181-912e-1a40f1725862/volumes/kubernetes.io~projected/kube-api-access-577tv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~projected/kube-api-access-8cfbt:{mountpoint:/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~projected/kube-api-access-8cfbt major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~projected/kube-api-access-74t85:{mountpoint:/var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~projected/kube-api-access-74t85 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fee1f49a-b7d6-46f9-b402-86429c1be8e2/volumes/kubernetes.io~projected/kube-api-access-gj8cv:{mountpoint:/var/lib/kubelet/pods/fee1f49a-b7d6-46f9-b402-86429c1be8e2/volumes/kubernetes.io~projected/kube-api-access-gj8cv major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/801e23a7bd80f834dfe0d130774df54c22d21ece4470c39dcd714a358aa3b675/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/521ffdac42d26bc8ac336619aa2aa3fef67f7d8e70b2dfb1dd15d0a7501ede2c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/405678e6a02ea9517c18ec7796a7be8ba41985d321e88cbdee106c8012f2bacd/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7225c15a7e6b905c9fc9192547da2042c38f7edb6ae91a08bc4211fc5210a701/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/a50e9e91a7e6a26bff2d55b8c1feb0345db521d836ffcfdc15ab43bb6dec9848/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d3d70b929abaeae0ecf993d95432d800af6fe2978ea3f3dc99e825ef0ab4871f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e73e6d1055c86207a4b4ee77ca756b7d7b69738e282dc19611061b6eeaac161d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4b52132cfa3ddb56725060742227d16d13fab1cd7f989200354fac7e80fc9700/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/403951580e92819bcbde951f77b52b3dd17887aaceae1e31f94b8ab478605fa0/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/6db6e889309435c899169d6aa9426d3c442058a5211da7265f5a1453c47e108b/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a77235c36df1a1a9b532d51df3557c267f8303af0c451efd123590f240f7d467/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b28365d9d2a8cbaf33978e38eaf787cc77a10e55c5c96f1ce4eac922a75e33cb/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e6cb9b3664aa7748518e38d111e54691a8bf9ef7ec5ee127d189cc6523bf03f6/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/70bf4e6bb9adf1b6336d25651ddbc340f2b3790e79d78ee3244555bb295cf972/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/68325c168144a8743c764bef19b6aeca78bff9cc6b16eb9b4ae63051c0812fa5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1a7aa969b612112d98ae81526288b42ad7274aff66ba86859a40c8db9d176344/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d6f87a265e75889f7fc19d270576bf59ef93d58f1653738f71f10dc8b21b94ed/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4aff1da2ef315bf05eadd6e067301d7d476660fa328d8dd28f30c419f386d03e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/93ca74ff14f68426e80022d144835786b7de5b9c7b3881c8c0db2cb73809840d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/9c1ccccce47c2c79a15438d04ca7237303762b631653e1b1321206a9b6cb7c4a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/766d7b01a0a8c5cdd494d95262e952cff0290311af3ec7900f1b298fad39b82c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/858fa06aeb88a11b76b58b925fbbf22b0c684d673990ef5f55eec12e865c3ac6/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/750461eeb717f78d3659776c60da84c61215492091d73509f9d2c14de2d7f029/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/99b110e16705efaa7e7734b4022ca2f28af617a2aec1825fc5e013952d7ec662/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/45ce3ee3bfce8f494e22265fc0a80b684c0d9b55114e5a9f1aa0b988859232f8/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/87b780bff035fc3c29c78cc5d3c05228bba8b92768848cd3cab084f30dfd4f30/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/e8f0c16c1b9b6f2dc224b431f4e9de7a67a078b60105814a72fbc5526d20c7ec/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/0965aa98b5d0178320367b3227cfe77903758ac980d9cbc3421d9a423f486038/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/f8700634a23fc82f4e6e98fcc2d8c3d017253b2485d2219251e6e9f0e0705b49/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/5f5c38670fc54e22b7776454d0cdf052518ac74f848d2aaba220f8fd6aaee567/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/3c6a05b7c37693eea05bbab4180dbd7bef080ca7b871ff8f185372d2bce810fd/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/f690548e9bbb8417e9a9e9a87cb2344c1758d5f2841d11c0d8f4c10f5dd792e3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/c21be69ba3aded845f2b41cd9939f33b69ab525a4bb61832af46c2eec8d203de/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/c029e7064cb92d26745c35db3f3d905d9af5e184d9b2a2451336906aa3f49be0/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/3f78c3a17b908d99058ecf070beecdb837c61bc82a76333fcf5a9e6aa9fe5853/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/11c31030c66476d1ecdd9552a1a74409fc6c1c3687155445f49cdcc02f2e335a/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/092b6f1975ceb4c670071cd7aee7a72a1cf29bba5fb6bd8241fb3a3516fdfa6b/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/2113d3a4c4e904edcceffd0af767e789928a804a6d4d40c985846292b6a15389/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/bf2371da602963a2f2a0800e6b04e69b02d94222b523d6beeb6e1ae73c948de5/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/e06a1a1cab7684dd1ad16e4dc88a18787de9bad2dd24da17ce1532b317e17cf6/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/d2045cda1feaaa402c54a8c7b66d8b9dd1f7de5e844505afafad7c189705cffb/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/005bfaf3fc11f399aaaec9d9444334968e8ecb058aec7b8bb85d255fbea9e316/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/7ccb6207b99c389eed7f4a3f117710155ddb1274295548d4dd7bcbb753a3bba8/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/8be9bcfe55eca54eb5449d622304495fb3f70b3423d627a832376f97832a61f8/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/616de9f1187f3fb9c916add662c865a4388357aa69a1c351dadc0047a76c8060/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/5ac5211753bcd41ec989847c35344ac13f46f5ff91a1517fd8b05b257bf774eb/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/ff5ed962488d4fd4398186e79bd37136aa626915b046dc8faea77b6bfece2541/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/18d978e7a6d080f9604c8d96828d4ebe0924bd2e676993b63a368489a756eb30/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/e416efebf7842a6cf18968f10a662ccd221cc21171dd113ef19d87f3efb9caa2/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/2782cf10edc7ead98f15ee0283f5b4c52096da7eddaa989ecb316b0ee1d4a807/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/0a2b4c8baae254630d3c64e44485bd2ce3d232be7d41737031217919b8f9810e/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/16e7c5611c2ce16fca53acaff16b6b051b09bb783fd20a70c01ca40b9754504d/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/8ccef2bcf4d722641373f09620156c9c707609a139f176445a409bf9fb07ddbc/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/ba93c808b15b68c71217377f4b7f1f631e4f5dc68d0854d4450dba493f3e0c17/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/e2de15f3ad47e135103f53cafe32e6a6c384ced8e53c126b4bd675a87a965ed6/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/3c8f52de6c10a01ef0fbfe81132b8fff123a9551dee915dab99102dc6b6018f3/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/95a09a194f848746a7be85641471493ea475f6eda6fad99fc8cac3344dbad553/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/f1af64cb477113f8d0881a2029fac8706e2ab877f69a33718db55b06dfbbdfd3/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/8c4bb02cfd5b8bb26d0e6297103966b4933fd403f423acbad6225db1955e53c8/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/df8e40a7525baa69d83b5173a703d20ae4e0dc8c1ac135ec58d8185cd20d59eb/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/a0d7bc64ecba4cf418d613df66b310a0d4188b174d6446409e0000a26758ee18/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/42cd6d157fd5bf59f43c870acc4cbb394ec5afad51317797382281b8a260169d/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/3a1ae1cd080ef6e558f1909f7dc6714ab610a758cf4f1a035eed24459da41f2f/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/1ca05645898589984355814f553f85aec13d604bde6fc81a228b16caf0bad71c/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/a3c161d9b7c93421c791db102de3a2419225921792c5b7d32da7cd0562ee00b9/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/9e4079052dd0d2b846c5ad9664379d0da218e2f1f8a8f0bc90a377681b413715/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/91b0e300e0dd413523a961f60e5d688e8e3408b8a84f323cb1cfa808bd191a55/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-471:{mountpoint:/var/lib/containers/storage/overlay/6973350d0e6f6025029334fabae2cbb559755067c350eb580c2d8bfb36a5bcf7/merged major:0 minor:471 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/d0ab2372f2789762f864c4252fb5df25eed3e2448e014192bb3f8283ceb7ccee/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/226b261521f42ce3cba70b232c6aafc418886a3b4d8050c40faccd7eb992992b/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d8f95a82658010055e89c85957406fbd31d06b262504088572a40c7b56fd3656/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/c315a585b16d736010a0014a02c6d5d5d9deea2687d94135bdbce61b0d8bfa14/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a570365f8994d2a8168e6b958246e72fa2d945794de8ea530bb6240afd4164e7/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/29707f42e098aceff91559a6fc12dd0237f653095abcd75d52da4b615fa06baf/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9bb4cf4504794fa324c0705d9a535b8415e46a89c3644190fddcffa3bfd308c2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/99f6090748fa7302572179149972c53498accb7c018448f6e0b78ff678d86bd4/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6c1ae81bc20dbf2779253503cec07e8845e1a7f6f213a9f820ca1cc97ff93fc8/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/812b642d051dc8899956f91ef93c45df481b7161f10bfa4195f609df6578a5bc/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/dc36ce892707b04ac3e0ab8e2f9468a3a26f2d13e50526691891151da7c24b91/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/2f8531f7d310f97e2c582e73171d09c6c4b513885529eff8469cac494fcc4d34/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.255524 5756 manager.go:217] Machine: {Timestamp:2026-10-31 00:57:08.255044998 +0000 UTC m=+0.062012538 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097295 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec229d2cd4af3340095fc9ae053616c3 SystemUUID:ec229d2c-d4af-3340-095f-c9ae053616c3 BootID:33d6cd26-724d-4abb-9bd2-3cc8c8bdc862 Filesystems:[{Device:/var/lib/kubelet/pods/73310d59-0bee-4124-ad47-b480d3544468/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:397 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-471 DeviceMajor:0 DeviceMinor:471 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fee1f49a-b7d6-46f9-b402-86429c1be8e2/volumes/kubernetes.io~projected/kube-api-access-gj8cv DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volumes/kubernetes.io~projected/kube-api-access-vn28c DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73310d59-0bee-4124-ad47-b480d3544468/volumes/kubernetes.io~projected/kube-api-access-jf7mb DeviceMajor:0 DeviceMinor:395 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0fda59f9-289b-431d-ba24-c291169fe1ae/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:282 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:408 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55a814403175e675e353264909cf99f90797087e9024a90512c4c960e4be9141/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:337 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c7a037a63ca679cd95054d8621c01ef608d0da9b94ba49cf148aacaee14bc05/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:401 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/380815d509854a75c2c5b71641e85298d51ec52d3e2b3681b96dfce57d08eb57/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fd953319df67201f93a68e00efb2313fce90b0aeb7f800f82b96e2d5ab06d0b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:257 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a2da62bf5ca56ca75f55851dc18671c7d91ac5f3f5e5c06b4aabdd8bb3778ed/userdata/shm DeviceMajor:0 DeviceMinor:369 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~projected/kube-api-access-8cfbt DeviceMajor:0 DeviceMinor:411 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbaba560-5774-4a19-afde-447fb6548557/volumes/kubernetes.io~projected/kube-api-access-hl4lx DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~projected/kube-api-access-h2b8r DeviceMajor:0 DeviceMinor:244 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d7d37da6ffd36c74f11ece0ebe38221b28c8d045f1153f50e7d8f879c006c3e/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/373ac6604c5e90a2efcc76abcb0dab799e093e8a2156dc1ef9bccf2b288f188b/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec524da9e997dc7fa9e91f6971bc3975502c98516ec42cacacc4212862ca4764/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66bb82ee-c3c0-4719-8a69-04ac4ce085cd/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:277 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~projected/kube-api-access-2x8cf DeviceMajor:0 DeviceMinor:298 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/10f64c92-95d2-4fc4-add2-d174ec7fe24f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63419a67-aaca-408d-bcfa-7c302b749c9d/volumes/kubernetes.io~projected/kube-api-access-nzsz8 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:258 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:400 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0fda59f9-289b-431d-ba24-c291169fe1ae/volumes/kubernetes.io~projected/kube-api-access-gdhvt DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52c24dba4acf4702399e76ff15c6a9d40eae1ab60f13ea814472e60b1b8a63aa/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:364 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b74dd1a9b57e0e33783afb737ca9f976d259858b7e0301d6c0a811af84168890/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~projected/kube-api-access-kq5q4 DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d399b8b352ba3a370343b104ea89445952877b53f6c4b2b3f079deeff76cfc09/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21b4576f03de4006df74fb396cf426ff72ed9a905f3c5106a0fbda6ec98ddacb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64d66bc1-aa19-496e-82c5-ec3bedaa9943/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:365 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d090b9a-abb1-48c3-9a43-415165ea4c04/volumes/kubernetes.io~projected/kube-api-access-pphfn DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b6739cc2b9fb54f4367f99f27b2e813b4b82da46963514fb26311bcffef9b3c1/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:366 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01262e31-9355-45a2-adc6-2f7889c4fe8e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~projected/kube-api-access-xf7m2 DeviceMajor:0 DeviceMinor:402 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e951e6b3-30a9-4181-912e-1a40f1725862/volumes/kubernetes.io~projected/kube-api-access-577tv DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/41b41813-6309-49be-bd2c-76b0820899af/volumes/kubernetes.io~projected/kube-api-access-n5spv DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e54b61a01999115f07e037e08a81f256bba054d7ae722c6795c1f143b1e2049f/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~projected/kube-api-access-2n7pp DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a96a806813e3d7573a361d33d2af4d68f24b5338d3cfd5cb2b77046ab8e88ab0/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab1705b372ba0e376571484f5d74742bc81c5fd1035c28c3613231e216ffe125/userdata/shm DeviceMajor:0 DeviceMinor:396 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/236b6fa3-674c-4790-951e-8b91cb51deee/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:399 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/41b41813-6309-49be-bd2c-76b0820899af/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes/kubernetes.io~projected/kube-api-access-xkvz8 DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f7d41401880aea6b17cad926129a9115ae11155b1b4e619e104384ccbc3bb00/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cbaba560-5774-4a19-afde-447fb6548557/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6789291bd342514b681e5705e3164fdbf77da40509e345954f48f52271f0b5f3/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01262e31-9355-45a2-adc6-2f7889c4fe8e/volumes/kubernetes.io~projected/kube-api-access-2fcnq DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d8189bdc-c813-4a8d-a1e0-052a4415d4ba/volumes/kubernetes.io~projected/kube-api-access-58psx DeviceMajor:0 DeviceMinor:297 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~projected/kube-api-access-k8pwc DeviceMajor:0 DeviceMinor:373 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:294 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7796b4f171bb5a6ee62b40436698ad00b6ed9eac460d80f4ddc3824f2fb6045/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6aed36cdb18e3932c157dc7a78594811aad7d3214ee464397088bf22b727a241/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:363 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:368 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3537b0f9-a07f-423d-a9e2-8a3abc3c3916/volumes/kubernetes.io~projected/kube-api-access-4d2k2 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c49c201-987e-4ea9-9bec-343a09101685/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:367 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ff778a1-143c-471d-a387-3897770b9502/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~projected/kube-api-access-74t85 DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a62692fc-3f77-478d-90d5-577b92632a39/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73689eddd0d5afc85ef01bb682c8c82aabeaadae678caaf21070b13d7d4bc583/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:407 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1f47fdf3-f591-4184-abff-d4670f404937/volumes/kubernetes.io~projected/kube-api-access-w2lts DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc981b83-c624-4524-95c0-950ec6a283ee/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d25f0afe-e3c3-4ded-ab76-c98083607e36/volumes/kubernetes.io~projected/kube-api-access-qnx6j DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01976d13abaa2d1 MacAddress:36:d3:17:f0:32:cc Speed:10000 Mtu:8901} {Name:380815d509854a7 MacAddress:22:da:72:c7:75:e8 Speed:10000 Mtu:8901} {Name:4fd953319df6720 MacAddress:7e:b1:82:90:d7:63 Speed:10000 Mtu:8901} {Name:551b245f7d17b68 MacAddress:d6:e9:74:0e:7c:00 Speed:10000 Mtu:8901} {Name:6789291bd342514 MacAddress:86:94:ef:c8:0f:39 Speed:10000 Mtu:8901} {Name:6a2da62bf5ca56c MacAddress:36:1b:4c:1b:ce:a7 Speed:10000 Mtu:8901} {Name:6aed36cdb18e393 MacAddress:9e:6a:fa:6d:a2:2a Speed:10000 Mtu:8901} {Name:73689eddd0d5afc MacAddress:de:89:14:93:fe:80 Speed:10000 Mtu:8901} {Name:9d7d37da6ffd36c MacAddress:4a:4e:42:4b:6c:90 Speed:10000 Mtu:8901} {Name:a96a806813e3d75 MacAddress:aa:0f:8e:ae:86:ff Speed:10000 Mtu:8901} {Name:ab1705b372ba0e3 MacAddress:22:88:8d:5e:e6:36 Speed:10000 Mtu:8901} {Name:b74dd1a9b57e0e3 MacAddress:9a:07:f6:01:3a:44 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:24:7f:2d:01:35 Speed:0 Mtu:9001} {Name:br-int MacAddress:3a:ec:30:c8:e0:3a Speed:0 Mtu:8901} {Name:c94708bb314ddc8 MacAddress:ee:9e:25:73:e6:98 Speed:10000 Mtu:8901} {Name:e54b61a01999115 MacAddress:c2:a5:e5:c8:44:fb Speed:10000 Mtu:8901} {Name:ec524da9e997dc7 MacAddress:56:10:2c:d4:bd:50 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:24:7f:2d:01:35 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:5a:ec:7d:78:8d:e0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:12:d1:5f:c4:69:81 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.255956 5756 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.255999 5756 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256273 5756 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256289 5756 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-249.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256409 5756 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256416 5756 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256426 5756 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256445 5756 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256516 5756 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256687 5756 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256798 5756 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256833 5756 kubelet.go:493] "Attempting to sync node with API server" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256844 5756 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256853 5756 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256878 5756 kubelet.go:405] "Adding apiserver pod source" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.256890 5756 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.259126 5756 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:57:08.259730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.259148 5756 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:57:08.260489 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.260056 5756 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 00:57:08.263598 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.263584 5756 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 00:57:08.263640 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.263600 5756 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 00:57:08.263706 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.263696 5756 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 00:57:08.264039 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264026 5756 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 00:57:08.264072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264058 5756 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 00:57:08.264157 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264148 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 00:57:08.264197 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264189 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 00:57:08.264225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264200 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 00:57:08.264225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264205 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 00:57:08.264225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264211 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 00:57:08.264225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264217 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 00:57:08.264225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264225 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264235 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264245 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264252 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264265 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264274 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264297 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 00:57:08.264396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.264307 5756 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 00:57:08.267677 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.267664 5756 server.go:1263] "Started kubelet" Oct 31 00:57:08.267780 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.267750 5756 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 00:57:08.268171 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.268119 5756 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 00:57:08.268264 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.268190 5756 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 00:57:08.268246 ip-10-0-136-249 systemd[1]: Started Kubernetes Kubelet. Oct 31 00:57:08.269651 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.269454 5756 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 00:57:08.270250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.269880 5756 server.go:317] "Adding debug handlers to kubelet server" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.273164 5756 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.274295 5756 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.274309 5756 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.274421 5756 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.274298 5756 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.274552 5756 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.274918 5756 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:50:56 +0000 UTC" deadline="2028-04-02 20:03:14.04028408 +0000 UTC" Oct 31 00:57:08.276003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.274958 5756 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12475h6m5.765330519s" Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276352 5756 factory.go:55] Registering systemd factory Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276379 5756 factory.go:223] Registration of the systemd container factory successfully Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276633 5756 factory.go:153] Registering CRI-O factory Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276649 5756 factory.go:223] Registration of the crio container factory successfully Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276730 5756 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276750 5756 factory.go:103] Registering Raw factory Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.276902 5756 manager.go:1199] Started watching for new ooms in manager Oct 31 00:57:08.278246 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.277309 5756 manager.go:319] Starting recovery of all containers Oct 31 00:57:08.279126 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279089 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41b41813-6309-49be-bd2c-76b0820899af" volumeName="kubernetes.io/secret/41b41813-6309-49be-bd2c-76b0820899af-metrics-tls" seLinuxMountContext="" Oct 31 00:57:08.279219 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279134 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d66bc1-aa19-496e-82c5-ec3bedaa9943" volumeName="kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-tls" seLinuxMountContext="" Oct 31 00:57:08.279219 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279149 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config" seLinuxMountContext="" Oct 31 00:57:08.279219 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279162 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp" seLinuxMountContext="" Oct 31 00:57:08.279219 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279181 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume" seLinuxMountContext="" Oct 31 00:57:08.279219 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279193 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp" seLinuxMountContext="" Oct 31 00:57:08.279219 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279212 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fda59f9-289b-431d-ba24-c291169fe1ae" volumeName="kubernetes.io/projected/0fda59f9-289b-431d-ba24-c291169fe1ae-kube-api-access-gdhvt" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279225 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f47fdf3-f591-4184-abff-d4670f404937" volumeName="kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279246 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41b41813-6309-49be-bd2c-76b0820899af" volumeName="kubernetes.io/configmap/41b41813-6309-49be-bd2c-76b0820899af-config-volume" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279259 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d66bc1-aa19-496e-82c5-ec3bedaa9943" volumeName="kubernetes.io/projected/64d66bc1-aa19-496e-82c5-ec3bedaa9943-kube-api-access-h2b8r" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279272 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73310d59-0bee-4124-ad47-b480d3544468" volumeName="kubernetes.io/projected/73310d59-0bee-4124-ad47-b480d3544468-kube-api-access-jf7mb" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279289 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2" volumeName="kubernetes.io/empty-dir/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-volume-directive-shadow" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279302 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62692fc-3f77-478d-90d5-577b92632a39" volumeName="kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279320 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62692fc-3f77-478d-90d5-577b92632a39" volumeName="kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279338 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279356 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279370 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279382 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279398 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbaba560-5774-4a19-afde-447fb6548557" volumeName="kubernetes.io/empty-dir/cbaba560-5774-4a19-afde-447fb6548557-data-volume" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279410 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279429 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279441 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc981b83-c624-4524-95c0-950ec6a283ee" volumeName="kubernetes.io/projected/fc981b83-c624-4524-95c0-950ec6a283ee-kube-api-access-74t85" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279454 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63419a67-aaca-408d-bcfa-7c302b749c9d" volumeName="kubernetes.io/configmap/63419a67-aaca-408d-bcfa-7c302b749c9d-serviceca" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279470 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2" volumeName="kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-metrics-client-ca" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279482 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out" seLinuxMountContext="" Oct 31 00:57:08.279534 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279500 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e951e6b3-30a9-4181-912e-1a40f1725862" volumeName="kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-multus-daemon-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279521 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279570 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-grpc-tls" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279601 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279614 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3537b0f9-a07f-423d-a9e2-8a3abc3c3916" volumeName="kubernetes.io/projected/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-kube-api-access-4d2k2" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279632 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279646 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279661 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279678 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d090b9a-abb1-48c3-9a43-415165ea4c04" volumeName="kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-tmp" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279693 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41b41813-6309-49be-bd2c-76b0820899af" volumeName="kubernetes.io/projected/41b41813-6309-49be-bd2c-76b0820899af-kube-api-access-n5spv" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279734 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ff778a1-143c-471d-a387-3897770b9502" volumeName="kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279774 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279801 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/projected/236b6fa3-674c-4790-951e-8b91cb51deee-kube-api-access-xf7m2" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279816 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279830 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25f0afe-e3c3-4ded-ab76-c98083607e36" volumeName="kubernetes.io/configmap/d25f0afe-e3c3-4ded-ab76-c98083607e36-iptables-alerter-script" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279847 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc981b83-c624-4524-95c0-950ec6a283ee" volumeName="kubernetes.io/configmap/fc981b83-c624-4524-95c0-950ec6a283ee-metrics-client-ca" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279887 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01262e31-9355-45a2-adc6-2f7889c4fe8e" volumeName="kubernetes.io/secret/01262e31-9355-45a2-adc6-2f7889c4fe8e-cert" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279907 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-tls" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279920 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/configmap/2c49c201-987e-4ea9-9bec-343a09101685-metrics-client-ca" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279938 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62692fc-3f77-478d-90d5-577b92632a39" volumeName="kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-metrics-client-ca" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279958 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279971 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.279989 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280003 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41b41813-6309-49be-bd2c-76b0820899af" volumeName="kubernetes.io/empty-dir/41b41813-6309-49be-bd2c-76b0820899af-tmp-dir" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280016 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2" volumeName="kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280033 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2" volumeName="kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-tls" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280048 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2" volumeName="kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280065 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62692fc-3f77-478d-90d5-577b92632a39" volumeName="kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-tls" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280079 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e951e6b3-30a9-4181-912e-1a40f1725862" volumeName="kubernetes.io/projected/e951e6b3-30a9-4181-912e-1a40f1725862-kube-api-access-577tv" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280093 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f47fdf3-f591-4184-abff-d4670f404937" volumeName="kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280111 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280147 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/empty-dir/2c49c201-987e-4ea9-9bec-343a09101685-tmp" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280165 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/projected/2c49c201-987e-4ea9-9bec-343a09101685-kube-api-access-k8pwc" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280178 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73310d59-0bee-4124-ad47-b480d3544468" volumeName="kubernetes.io/secret/73310d59-0bee-4124-ad47-b480d3544468-monitoring-plugin-cert" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280190 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280208 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280220 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25f0afe-e3c3-4ded-ab76-c98083607e36" volumeName="kubernetes.io/projected/d25f0afe-e3c3-4ded-ab76-c98083607e36-kube-api-access-qnx6j" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280237 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f64c92-95d2-4fc4-add2-d174ec7fe24f" volumeName="kubernetes.io/configmap/10f64c92-95d2-4fc4-add2-d174ec7fe24f-konnectivity-ca" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280251 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280263 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62692fc-3f77-478d-90d5-577b92632a39" volumeName="kubernetes.io/empty-dir/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-textfile" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280280 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280293 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e951e6b3-30a9-4181-912e-1a40f1725862" volumeName="kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-cni-binary-copy" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280310 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280323 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280348 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fee1f49a-b7d6-46f9-b402-86429c1be8e2" volumeName="kubernetes.io/projected/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kube-api-access-gj8cv" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280365 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-client-ca-bundle" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280378 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280396 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc981b83-c624-4524-95c0-950ec6a283ee" volumeName="kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280408 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc981b83-c624-4524-95c0-950ec6a283ee" volumeName="kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280420 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fda59f9-289b-431d-ba24-c291169fe1ae" volumeName="kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs" seLinuxMountContext="" Oct 31 00:57:08.280415 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280436 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f47fdf3-f591-4184-abff-d4670f404937" volumeName="kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-binary-copy" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280448 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/empty-dir/236b6fa3-674c-4790-951e-8b91cb51deee-audit-log" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280464 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c49c201-987e-4ea9-9bec-343a09101685" volumeName="kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280478 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d090b9a-abb1-48c3-9a43-415165ea4c04" volumeName="kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-tuned" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280491 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ff778a1-143c-471d-a387-3897770b9502" volumeName="kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-script-lib" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280506 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2" volumeName="kubernetes.io/projected/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-api-access-kq5q4" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280518 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a62692fc-3f77-478d-90d5-577b92632a39" volumeName="kubernetes.io/projected/a62692fc-3f77-478d-90d5-577b92632a39-kube-api-access-2x8cf" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280535 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01262e31-9355-45a2-adc6-2f7889c4fe8e" volumeName="kubernetes.io/projected/01262e31-9355-45a2-adc6-2f7889c4fe8e-kube-api-access-2fcnq" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280548 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="236b6fa3-674c-4790-951e-8b91cb51deee" volumeName="kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-metrics-server-audit-profiles" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280561 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ff778a1-143c-471d-a387-3897770b9502" volumeName="kubernetes.io/projected/4ff778a1-143c-471d-a387-3897770b9502-kube-api-access-vn28c" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280579 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63419a67-aaca-408d-bcfa-7c302b749c9d" volumeName="kubernetes.io/projected/63419a67-aaca-408d-bcfa-7c302b749c9d-kube-api-access-nzsz8" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280592 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d66bc1-aa19-496e-82c5-ec3bedaa9943" volumeName="kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280610 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66bb82ee-c3c0-4719-8a69-04ac4ce085cd" volumeName="kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280624 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280637 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b2593841-e6ff-4bce-a8d8-a8161a7b895c" volumeName="kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280655 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1f47fdf3-f591-4184-abff-d4670f404937" volumeName="kubernetes.io/projected/1f47fdf3-f591-4184-abff-d4670f404937-kube-api-access-w2lts" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280680 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d090b9a-abb1-48c3-9a43-415165ea4c04" volumeName="kubernetes.io/projected/3d090b9a-abb1-48c3-9a43-415165ea4c04-kube-api-access-pphfn" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280698 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280712 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbaba560-5774-4a19-afde-447fb6548557" volumeName="kubernetes.io/projected/cbaba560-5774-4a19-afde-447fb6548557-kube-api-access-hl4lx" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280724 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbaba560-5774-4a19-afde-447fb6548557" volumeName="kubernetes.io/secret/cbaba560-5774-4a19-afde-447fb6548557-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280742 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8189bdc-c813-4a8d-a1e0-052a4415d4ba" volumeName="kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280756 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3d183d8-26d7-4e77-aa32-64e75fc435a5" volumeName="kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280774 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10f64c92-95d2-4fc4-add2-d174ec7fe24f" volumeName="kubernetes.io/secret/10f64c92-95d2-4fc4-add2-d174ec7fe24f-agent-certs" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280787 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3537b0f9-a07f-423d-a9e2-8a3abc3c3916" volumeName="kubernetes.io/empty-dir/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-tmp-dir" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280801 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ff778a1-143c-471d-a387-3897770b9502" volumeName="kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-env-overrides" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280819 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ff778a1-143c-471d-a387-3897770b9502" volumeName="kubernetes.io/secret/4ff778a1-143c-471d-a387-3897770b9502-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280833 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64d66bc1-aa19-496e-82c5-ec3bedaa9943" volumeName="kubernetes.io/configmap/64d66bc1-aa19-496e-82c5-ec3bedaa9943-metrics-client-ca" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280851 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" volumeName="kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280883 5756 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbaba560-5774-4a19-afde-447fb6548557" volumeName="kubernetes.io/configmap/cbaba560-5774-4a19-afde-447fb6548557-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280895 5756 reconstruct.go:97] "Volume reconstruction finished" Oct 31 00:57:08.284135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.280903 5756 reconciler.go:29] "Reconciler: start to sync state" Oct 31 00:57:08.301233 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.301191 5756 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 00:57:08.305080 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.303307 5756 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 00:57:08.305080 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.303337 5756 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 00:57:08.305080 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.303365 5756 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 00:57:08.305080 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.303419 5756 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 00:57:08.324517 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.324483 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" exitCode=0 Oct 31 00:57:08.354506 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.354471 5756 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="10aaf6d46e7b0f9e66d0b3e1f846fd7b2255ff2f28a789519b147cc6f562802e" exitCode=0 Oct 31 00:57:08.354506 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.354503 5756 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="7abb589564d9fff8b2d6a2db95f1a32c958eacd32ffd3501bafdbcf105681373" exitCode=0 Oct 31 00:57:08.354680 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.354514 5756 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="ca2e35e8d45c3b680ad98fd62f743af5763d0930945785179a3f793a04a2e1f8" exitCode=0 Oct 31 00:57:08.354680 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.354524 5756 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="6b3b487b8c1b6e6355e072675c11419fffe7bfdf71118fb3b33fa9193b2c0113" exitCode=0 Oct 31 00:57:08.354680 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.354533 5756 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="c400bf6a680156257533eb3f7d976ac66d89c2ea26d82cb1d3c2338ccb287558" exitCode=0 Oct 31 00:57:08.354680 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.354540 5756 generic.go:356] "Generic (PLEG): container finished" podID="1f47fdf3-f591-4184-abff-d4670f404937" containerID="34b7f8ed05253d37880c287ba7c138eafa6eff49b7562115a65742c9dde97e50" exitCode=0 Oct 31 00:57:08.357900 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.357853 5756 generic.go:356] "Generic (PLEG): container finished" podID="7b884c31d96cb5a9a71220f2bda573a2" containerID="f0b3c575252f55f2d02bdb3900650d7693c8f478ef58ea543810b1d982f556b1" exitCode=0 Oct 31 00:57:08.361980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.361961 5756 generic.go:356] "Generic (PLEG): container finished" podID="a62692fc-3f77-478d-90d5-577b92632a39" containerID="846b3a840bb9e1237c381a4f1a223aff4197a3c0ee8e1dd2688757d781fd4ca5" exitCode=0 Oct 31 00:57:08.374598 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.374574 5756 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:57:08.403528 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.403508 5756 manager.go:324] Recovery completed Oct 31 00:57:08.403955 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.403932 5756 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 00:57:08.425340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.425319 5756 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:57:08.427178 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.427160 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:57:08.427275 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.427193 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:57:08.427275 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.427206 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:57:08.430120 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430102 5756 cpu_manager.go:226] "Starting" policy="none" Oct 31 00:57:08.430169 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430119 5756 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 00:57:08.430169 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430125 5756 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:57:08.430169 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430139 5756 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 00:57:08.430298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430283 5756 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 00:57:08.430331 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430300 5756 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 00:57:08.430331 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430317 5756 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 00:57:08.430331 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430322 5756 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 00:57:08.430331 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430328 5756 policy_none.go:50] "Start" Oct 31 00:57:08.430440 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430337 5756 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 00:57:08.430440 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430347 5756 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 00:57:08.430496 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430477 5756 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 00:57:08.430496 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430484 5756 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 00:57:08.430496 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430489 5756 policy_none.go:44] "Start" Oct 31 00:57:08.430980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430966 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:57:08.431019 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.430999 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:57:08.431019 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.431015 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:57:08.435304 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.435280 5756 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 00:57:08.437838 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.437826 5756 manager.go:341] "Starting Device Plugin manager" Oct 31 00:57:08.437900 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.437850 5756 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 00:57:08.437900 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.437880 5756 server.go:87] "Starting device plugin registration server" Oct 31 00:57:08.438204 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438192 5756 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 00:57:08.438235 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438208 5756 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 00:57:08.438289 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438272 5756 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 00:57:08.438404 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438373 5756 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 00:57:08.438480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438414 5756 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 00:57:08.438480 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438420 5756 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 00:57:08.438573 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.438485 5756 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:57:08.438402913Z","UUID":"5645bb9c-da63-4387-bd71-e19cf7ddbed6","Handler":null,"Name":"","Endpoint":""} Oct 31 00:57:08.440375 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.440341 5756 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 00:57:08.440463 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.440393 5756 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-249.us-west-1.compute.internal\" not found" Oct 31 00:57:08.440777 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.440759 5756 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 00:57:08.440777 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.440779 5756 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 00:57:08.471350 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.471321 5756 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:57:08.471350 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.471327 5756 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:57:08.471523 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.471435 5756 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 00:57:08.472737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.472699 5756 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:57:08.538588 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.538556 5756 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 00:57:08.539649 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.539626 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 00:57:08.539789 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.539665 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 00:57:08.539789 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.539680 5756 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-249.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 00:57:08.539789 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.539745 5756 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.570521 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.570492 5756 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 00:57:08.604507 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.604463 5756 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal"] Oct 31 00:57:08.605151 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605117 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"e548c58b36505791ab3955482325fe49","Type":"ContainerStarted","Data":"e7437648e7b6143c4be811f5ccaf8151d8202903e98f20827e44d0b731945fc9"} Oct 31 00:57:08.605211 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605162 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"e548c58b36505791ab3955482325fe49","Type":"ContainerStarted","Data":"dcda0f3f894fffdc9ba767b86c20f45aa9e17d980e8008dba7c66fecb66f96c4"} Oct 31 00:57:08.605211 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605183 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"7b884c31d96cb5a9a71220f2bda573a2","Type":"ContainerStarted","Data":"4413cb6cb647eb0c70a671b52dd4e8349827d9364df119dd3afdef30ce8a27a5"} Oct 31 00:57:08.605211 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605197 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"7b884c31d96cb5a9a71220f2bda573a2","Type":"ContainerDied","Data":"f0b3c575252f55f2d02bdb3900650d7693c8f478ef58ea543810b1d982f556b1"} Oct 31 00:57:08.605296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605218 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" event={"ID":"7b884c31d96cb5a9a71220f2bda573a2","Type":"ContainerStarted","Data":"858e259b31e9613f18acc565fdd43e7014df337fed5358decbbd4cba5a48fb2d"} Oct 31 00:57:08.605296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605218 5756 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.605296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.605262 5756 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.648691 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.648667 5756 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.648841 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.648733 5756 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.666931 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.666910 5756 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:57:08.667020 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.666952 5756 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.667329 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.667314 5756 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:57:08.667360 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:08.667351 5756 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.773431 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.773351 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.773431 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.773381 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.773431 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.773398 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e548c58b36505791ab3955482325fe49-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"e548c58b36505791ab3955482325fe49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.875328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.874436 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.875328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.874487 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.875328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.874512 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e548c58b36505791ab3955482325fe49-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"e548c58b36505791ab3955482325fe49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.875328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.874569 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e548c58b36505791ab3955482325fe49-config\") pod \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"e548c58b36505791ab3955482325fe49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.875328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.874630 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:08.875328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:08.874663 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b884c31d96cb5a9a71220f2bda573a2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" (UID: \"7b884c31d96cb5a9a71220f2bda573a2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:09.257626 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.257595 5756 apiserver.go:52] "Watching apiserver" Oct 31 00:57:09.361430 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.361375 5756 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 00:57:09.363725 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.363683 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l72fc","openshift-network-diagnostics/network-check-target-blwrh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal","openshift-monitoring/metrics-server-6cf56d4544-tzfbs","openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7","openshift-ovn-kubernetes/ovnkube-node-kmmnx","openshift-ingress-canary/ingress-canary-gp5rq","openshift-network-operator/iptables-alerter-nzlwl","kube-system/global-pull-secret-syncer-48ddr","kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal","openshift-dns/node-resolver-649pf","openshift-image-registry/node-ca-vrqjp","openshift-insights/insights-runtime-extractor-c7pm5","openshift-cluster-node-tuning-operator/tuned-gbk88","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-69f8cc8d77-825m2","openshift-multus/multus-additional-cni-plugins-ztqdl","kube-system/konnectivity-agent-lttht","openshift-console/console-698867449c-xbf2l","openshift-dns/dns-default-wklrq","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-pgqj5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx","openshift-console/console-7c6444d66d-l4zqt","openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk","openshift-multus/multus-zdntg","openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs","openshift-monitoring/node-exporter-qw74k"] Oct 31 00:57:09.376311 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376282 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.376459 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376320 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-modprobe-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.376619 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376597 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pphfn\" (UniqueName: \"kubernetes.io/projected/3d090b9a-abb1-48c3-9a43-415165ea4c04-kube-api-access-pphfn\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.376678 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376640 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-bin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.376727 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376700 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-hostroot\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.376774 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376762 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-tls\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.376834 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376788 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-tmp-dir\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.376920 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376842 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-tmp-dir\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.376920 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376882 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-var-lib-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.376920 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376912 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff778a1-143c-471d-a387-3897770b9502-ovn-node-metrics-cert\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376927 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376943 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-k8s-cni-cncf-io\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376958 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-dbus\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376972 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-textfile\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.376987 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysconfig\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377011 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-kubelet\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.377049 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377044 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-netns\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.377290 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377063 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-multus\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.377290 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377082 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-textfile\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.377290 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377180 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-sys\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.377290 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377206 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-kubernetes\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.377290 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377282 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lts\" (UniqueName: \"kubernetes.io/projected/1f47fdf3-f591-4184-abff-d4670f404937-kube-api-access-w2lts\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.377468 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377314 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-tuned\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.377468 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377369 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-tuned\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.377468 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377417 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-etc-kubernetes\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.377560 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377510 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-run\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.377560 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377527 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8cv\" (UniqueName: \"kubernetes.io/projected/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kube-api-access-gj8cv\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.377560 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377543 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d2k2\" (UniqueName: \"kubernetes.io/projected/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-kube-api-access-4d2k2\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.377560 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377557 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-etc-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.377728 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377608 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b41813-6309-49be-bd2c-76b0820899af-tmp-dir\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.377728 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377650 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-config\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.377728 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377660 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b41813-6309-49be-bd2c-76b0820899af-tmp-dir\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.377728 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377678 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-env-overrides\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.377728 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377703 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbaba560-5774-4a19-afde-447fb6548557-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377731 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377776 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377803 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377833 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377890 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377919 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-sys\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377945 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377979 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.377993 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.377976 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-conf-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378017 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-kubelet-config\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378047 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378069 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-conf\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378102 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64d66bc1-aa19-496e-82c5-ec3bedaa9943-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378124 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63419a67-aaca-408d-bcfa-7c302b749c9d-host\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378142 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378184 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-cnibin\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378233 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-lib-modules\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378261 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc981b83-c624-4524-95c0-950ec6a283ee-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378278 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbaba560-5774-4a19-afde-447fb6548557-crio-socket\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378292 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-hosts-file\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378306 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-node-log\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378335 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-root\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378360 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-netns\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378398 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378421 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-kubelet\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378464 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-device-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378487 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41b41813-6309-49be-bd2c-76b0820899af-config-volume\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378510 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41b41813-6309-49be-bd2c-76b0820899af-metrics-tls\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378534 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.378577 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378559 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25f0afe-e3c3-4ded-ab76-c98083607e36-iptables-alerter-script\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378582 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25f0afe-e3c3-4ded-ab76-c98083607e36-host-slash\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378627 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-metrics-client-ca\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378650 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-systemd\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378675 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl4lx\" (UniqueName: \"kubernetes.io/projected/cbaba560-5774-4a19-afde-447fb6548557-kube-api-access-hl4lx\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378701 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-system-cni-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378766 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378827 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378852 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5spv\" (UniqueName: \"kubernetes.io/projected/41b41813-6309-49be-bd2c-76b0820899af-kube-api-access-n5spv\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378925 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01262e31-9355-45a2-adc6-2f7889c4fe8e-cert\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.378997 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2b8r\" (UniqueName: \"kubernetes.io/projected/64d66bc1-aa19-496e-82c5-ec3bedaa9943-kube-api-access-h2b8r\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379018 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379034 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-cni-binary-copy\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379049 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fcnq\" (UniqueName: \"kubernetes.io/projected/01262e31-9355-45a2-adc6-2f7889c4fe8e-kube-api-access-2fcnq\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379066 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379132 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379187 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-multus-certs\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379225 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8cf\" (UniqueName: \"kubernetes.io/projected/a62692fc-3f77-478d-90d5-577b92632a39-kube-api-access-2x8cf\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379323 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-systemd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379359 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-ovn\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379375 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379391 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.379443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379427 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379477 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379499 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10f64c92-95d2-4fc4-add2-d174ec7fe24f-agent-certs\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379516 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379552 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-bin\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379573 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74t85\" (UniqueName: \"kubernetes.io/projected/fc981b83-c624-4524-95c0-950ec6a283ee-kube-api-access-74t85\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379592 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-multus-daemon-config\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379687 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379726 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-registration-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379746 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-script-lib\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379769 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnx6j\" (UniqueName: \"kubernetes.io/projected/d25f0afe-e3c3-4ded-ab76-c98083607e36-kube-api-access-qnx6j\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379786 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq5q4\" (UniqueName: \"kubernetes.io/projected/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-api-access-kq5q4\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379801 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-cnibin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379815 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-host\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379909 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379955 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-system-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.379980 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-socket-dir-parent\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380008 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-wtmp\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380043 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-var-lib-kubelet\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380068 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsz8\" (UniqueName: \"kubernetes.io/projected/63419a67-aaca-408d-bcfa-7c302b749c9d-kube-api-access-nzsz8\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380084 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380175 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380204 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380226 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380244 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-tmp\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380263 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-netd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.380298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380297 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbaba560-5774-4a19-afde-447fb6548557-data-volume\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380323 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbaba560-5774-4a19-afde-447fb6548557-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380397 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cbaba560-5774-4a19-afde-447fb6548557-data-volume\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380414 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380437 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-sys-fs\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380434 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d090b9a-abb1-48c3-9a43-415165ea4c04-tmp\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380467 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-log-socket\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380488 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn28c\" (UniqueName: \"kubernetes.io/projected/4ff778a1-143c-471d-a387-3897770b9502-kube-api-access-vn28c\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380507 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380523 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-socket-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380539 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63419a67-aaca-408d-bcfa-7c302b749c9d-serviceca\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380554 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380597 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380675 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577tv\" (UniqueName: \"kubernetes.io/projected/e951e6b3-30a9-4181-912e-1a40f1725862-kube-api-access-577tv\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380724 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380749 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380767 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10f64c92-95d2-4fc4-add2-d174ec7fe24f-konnectivity-ca\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380783 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhvt\" (UniqueName: \"kubernetes.io/projected/0fda59f9-289b-431d-ba24-c291169fe1ae-kube-api-access-gdhvt\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380807 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-systemd-units\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.381250 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.380838 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-slash\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.381792 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.381775 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.388207 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.388162 5756 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:09.390732 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.390707 5756 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:09.417349 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417323 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-98n4j\"" Oct 31 00:57:09.417570 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417554 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 00:57:09.417700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417683 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xr8df\"" Oct 31 00:57:09.417741 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417691 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 00:57:09.417741 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417707 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ndfs4\"" Oct 31 00:57:09.417811 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417748 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxv7v\"" Oct 31 00:57:09.417811 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.417791 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lhvrj\"" Oct 31 00:57:09.418325 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.418308 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 00:57:09.418401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.418373 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fda59f9-289b-431d-ba24-c291169fe1ae-metrics-certs\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:57:09.418401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.418393 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fksdd\"" Oct 31 00:57:09.419386 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419372 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 00:57:09.419386 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419382 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wfh75\"" Oct 31 00:57:09.419750 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419734 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.419811 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419706 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 00:57:09.419811 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419740 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.419811 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419765 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/10f64c92-95d2-4fc4-add2-d174ec7fe24f-agent-certs\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:57:09.419916 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419881 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 00:57:09.419955 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419922 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.419984 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.419957 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.420037 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420023 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 00:57:09.420069 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420050 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.420069 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420058 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.420144 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420127 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 00:57:09.420191 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420143 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 00:57:09.420191 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420174 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.420191 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420184 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6sjwh\"" Oct 31 00:57:09.420302 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420224 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hnbxm\"" Oct 31 00:57:09.420302 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420255 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 00:57:09.420302 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420257 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.420302 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420288 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.420485 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420332 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.420485 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420419 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 00:57:09.420485 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420473 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420485 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420508 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420511 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420517 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420547 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gjj6f\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420548 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g8n5b\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420571 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.420617 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420572 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 00:57:09.420824 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420622 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 00:57:09.420851 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420830 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-original-pull-secret\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.421009 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.420994 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/10f64c92-95d2-4fc4-add2-d174ec7fe24f-konnectivity-ca\") pod \"konnectivity-agent-lttht\" (UID: \"10f64c92-95d2-4fc4-add2-d174ec7fe24f\") " pod="kube-system/konnectivity-agent-lttht" Oct 31 00:57:09.427959 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.427934 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-env-overrides\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.428008 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.427966 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ff778a1-143c-471d-a387-3897770b9502-ovn-node-metrics-cert\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.428008 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.427947 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-config\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.429119 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.429102 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d25f0afe-e3c3-4ded-ab76-c98083607e36-iptables-alerter-script\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.430331 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.430313 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-cni-binary-copy\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.430410 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.430318 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e951e6b3-30a9-4181-912e-1a40f1725862-multus-daemon-config\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.430410 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.430319 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.430410 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.430358 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ff778a1-143c-471d-a387-3897770b9502-ovnkube-script-lib\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.430410 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.430358 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f47fdf3-f591-4184-abff-d4670f404937-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.454562 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.454537 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gr6qq\"" Oct 31 00:57:09.455286 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455260 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.455401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455320 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-sk5g6\"" Oct 31 00:57:09.455401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455353 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 00:57:09.455401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455352 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.455401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455377 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 00:57:09.455570 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455504 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.455570 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455533 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.455732 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455709 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 00:57:09.455834 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455768 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 00:57:09.455834 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455813 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m5b2l\"" Oct 31 00:57:09.455834 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455827 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rf2ln\"" Oct 31 00:57:09.456010 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.455893 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 00:57:09.458052 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.458035 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cbaba560-5774-4a19-afde-447fb6548557-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.459171 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.459154 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/41b41813-6309-49be-bd2c-76b0820899af-config-volume\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.459237 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.459178 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/41b41813-6309-49be-bd2c-76b0820899af-metrics-tls\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.461498 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.461482 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cbaba560-5774-4a19-afde-447fb6548557-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.461795 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.461777 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/63419a67-aaca-408d-bcfa-7c302b749c9d-serviceca\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.467780 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.467760 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 31 00:57:09.469208 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.469194 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.472283 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.472265 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4fw5b\"" Oct 31 00:57:09.475140 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.475114 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pphfn\" (UniqueName: \"kubernetes.io/projected/3d090b9a-abb1-48c3-9a43-415165ea4c04-kube-api-access-pphfn\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.475225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.475116 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8cv\" (UniqueName: \"kubernetes.io/projected/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kube-api-access-gj8cv\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.475512 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.475492 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lts\" (UniqueName: \"kubernetes.io/projected/1f47fdf3-f591-4184-abff-d4670f404937-kube-api-access-w2lts\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.475844 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.475830 5756 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 00:57:09.480772 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.480750 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58psx\" (UniqueName: \"kubernetes.io/projected/d8189bdc-c813-4a8d-a1e0-052a4415d4ba-kube-api-access-58psx\") pod \"network-check-target-blwrh\" (UID: \"d8189bdc-c813-4a8d-a1e0-052a4415d4ba\") " pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:09.481625 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481605 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.481689 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481635 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.481689 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481651 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.481689 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481673 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.481820 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481700 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-system-cni-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.481820 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481741 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.481820 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481771 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.481820 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481744 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-system-cni-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.481820 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481809 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481833 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-client-certs\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481834 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481849 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481931 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481970 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-multus-certs\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.481999 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-multus-certs\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482013 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-systemd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482044 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482035 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-ovn\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482054 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482077 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482089 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-systemd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482094 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73310d59-0bee-4124-ad47-b480d3544468-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482102 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-ovn\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482117 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482131 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482201 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482215 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-run-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482222 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pwc\" (UniqueName: \"kubernetes.io/projected/2c49c201-987e-4ea9-9bec-343a09101685-kube-api-access-k8pwc\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482242 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482263 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482293 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-bin\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482271 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-bin\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482334 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482357 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.482390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482394 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-registration-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482429 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482469 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-registration-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482470 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-tls\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482510 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-cnibin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482537 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-host\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482564 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482593 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482621 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-system-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482623 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-cnibin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482634 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-host\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482648 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482660 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-socket-dir-parent\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482698 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-system-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482706 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-socket-dir-parent\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482699 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-wtmp\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482740 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-var-lib-kubelet\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482745 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-wtmp\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482760 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-netd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482745 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d2k2\" (UniqueName: \"kubernetes.io/projected/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-kube-api-access-4d2k2\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482785 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482809 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-var-lib-kubelet\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482811 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482809 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-cni-netd\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482901 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482935 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482978 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-sys-fs\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482998 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-log-socket\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483015 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-client-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483025 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-sys-fs\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483031 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.482998 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483050 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-socket-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483063 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-log-socket\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.483072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483098 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-socket-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483099 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483132 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483150 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483181 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483225 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483232 5756 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483238 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483258 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:09.483278 5756 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483278 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-etc-selinux\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483293 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483332 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-systemd-units\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483373 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-slash\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483390 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-systemd-units\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483413 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483428 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-slash\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483516 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483642 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483659 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483675 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-modprobe-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483692 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483709 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483742 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483779 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483747 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-cni-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483808 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-bin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483857 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-modprobe-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483917 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-bin\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483930 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl4lx\" (UniqueName: \"kubernetes.io/projected/cbaba560-5774-4a19-afde-447fb6548557-kube-api-access-hl4lx\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483954 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-hostroot\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483992 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-hostroot\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.483987 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-var-lib-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484013 5756 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484031 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484035 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-var-lib-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:09.484054 5756 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-249.us-west-1.compute.internal" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484067 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7mb\" (UniqueName: \"kubernetes.io/projected/73310d59-0bee-4124-ad47-b480d3544468-kube-api-access-jf7mb\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484117 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-k8s-cni-cncf-io\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484143 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-dbus\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484170 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysconfig\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484222 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-kubelet\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484249 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-grpc-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484292 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-dbus\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484336 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-netns\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484294 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-netns\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484351 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysconfig\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484387 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-kubelet\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484391 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-multus\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484421 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-sys\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484451 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-kubernetes\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484458 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhvt\" (UniqueName: \"kubernetes.io/projected/0fda59f9-289b-431d-ba24-c291169fe1ae-kube-api-access-gdhvt\") pod \"network-metrics-daemon-pgqj5\" (UID: \"0fda59f9-289b-431d-ba24-c291169fe1ae\") " pod="openshift-multus/network-metrics-daemon-pgqj5" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484472 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-cni-multus\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484479 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484491 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577tv\" (UniqueName: \"kubernetes.io/projected/e951e6b3-30a9-4181-912e-1a40f1725862-kube-api-access-577tv\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484508 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484517 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-kubernetes\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484507 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-sys\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484535 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnx6j\" (UniqueName: \"kubernetes.io/projected/d25f0afe-e3c3-4ded-ab76-c98083607e36-kube-api-access-qnx6j\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484552 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c49c201-987e-4ea9-9bec-343a09101685-tmp\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484583 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-etc-kubernetes\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484609 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-run\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484609 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c49c201-987e-4ea9-9bec-343a09101685-tmp\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484630 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-etc-kubernetes\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.484604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484638 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-etc-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484661 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-run\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484669 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484674 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-etc-openvswitch\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484700 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484716 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn28c\" (UniqueName: \"kubernetes.io/projected/4ff778a1-143c-471d-a387-3897770b9502-kube-api-access-vn28c\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484727 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484756 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-run-k8s-cni-cncf-io\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484764 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-sys\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484797 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/236b6fa3-674c-4790-951e-8b91cb51deee-audit-log\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484811 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-sys\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484825 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484852 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-conf-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484898 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/236b6fa3-674c-4790-951e-8b91cb51deee-audit-log\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484919 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c49c201-987e-4ea9-9bec-343a09101685-metrics-client-ca\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484948 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-multus-conf-dir\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.484976 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdlc\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485002 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-kubelet-config\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485037 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485050 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/66bb82ee-c3c0-4719-8a69-04ac4ce085cd-kubelet-config\") pod \"global-pull-secret-syncer-48ddr\" (UID: \"66bb82ee-c3c0-4719-8a69-04ac4ce085cd\") " pod="kube-system/global-pull-secret-syncer-48ddr" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485056 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsz8\" (UniqueName: \"kubernetes.io/projected/63419a67-aaca-408d-bcfa-7c302b749c9d-kube-api-access-nzsz8\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485113 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-conf\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485188 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485199 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-d\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485242 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7m2\" (UniqueName: \"kubernetes.io/projected/236b6fa3-674c-4790-951e-8b91cb51deee-kube-api-access-xf7m2\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485268 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63419a67-aaca-408d-bcfa-7c302b749c9d-host\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485283 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-sysctl-conf\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485296 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485324 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485326 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/63419a67-aaca-408d-bcfa-7c302b749c9d-host\") pod \"node-ca-vrqjp\" (UID: \"63419a67-aaca-408d-bcfa-7c302b749c9d\") " pod="openshift-image-registry/node-ca-vrqjp" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485365 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-cnibin\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485392 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-lib-modules\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485407 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f47fdf3-f591-4184-abff-d4670f404937-cnibin\") pod \"multus-additional-cni-plugins-ztqdl\" (UID: \"1f47fdf3-f591-4184-abff-d4670f404937\") " pod="openshift-multus/multus-additional-cni-plugins-ztqdl" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485412 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-metrics-server-audit-profiles\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485453 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485486 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-lib-modules\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485490 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485520 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbaba560-5774-4a19-afde-447fb6548557-crio-socket\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485532 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5spv\" (UniqueName: \"kubernetes.io/projected/41b41813-6309-49be-bd2c-76b0820899af-kube-api-access-n5spv\") pod \"dns-default-wklrq\" (UID: \"41b41813-6309-49be-bd2c-76b0820899af\") " pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485547 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-hosts-file\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485550 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cbaba560-5774-4a19-afde-447fb6548557-crio-socket\") pod \"insights-runtime-extractor-c7pm5\" (UID: \"cbaba560-5774-4a19-afde-447fb6548557\") " pod="openshift-insights/insights-runtime-extractor-c7pm5" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485571 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-node-log\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485587 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-root\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485595 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3537b0f9-a07f-423d-a9e2-8a3abc3c3916-hosts-file\") pod \"node-resolver-649pf\" (UID: \"3537b0f9-a07f-423d-a9e2-8a3abc3c3916\") " pod="openshift-dns/node-resolver-649pf" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485603 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-netns\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485620 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485637 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-kubelet\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485643 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62692fc-3f77-478d-90d5-577b92632a39-root\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485649 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-host-run-netns\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485652 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-device-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485678 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485684 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fee1f49a-b7d6-46f9-b402-86429c1be8e2-device-dir\") pod \"aws-ebs-csi-driver-node-54jrx\" (UID: \"fee1f49a-b7d6-46f9-b402-86429c1be8e2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-54jrx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485646 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ff778a1-143c-471d-a387-3897770b9502-node-log\") pod \"ovnkube-node-kmmnx\" (UID: \"4ff778a1-143c-471d-a387-3897770b9502\") " pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485689 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e951e6b3-30a9-4181-912e-1a40f1725862-host-var-lib-kubelet\") pod \"multus-zdntg\" (UID: \"e951e6b3-30a9-4181-912e-1a40f1725862\") " pod="openshift-multus/multus-zdntg" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485698 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25f0afe-e3c3-4ded-ab76-c98083607e36-host-slash\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485741 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d25f0afe-e3c3-4ded-ab76-c98083607e36-host-slash\") pod \"iptables-alerter-nzlwl\" (UID: \"d25f0afe-e3c3-4ded-ab76-c98083607e36\") " pod="openshift-network-operator/iptables-alerter-nzlwl" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485762 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-systemd\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.486645 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.485821 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3d090b9a-abb1-48c3-9a43-415165ea4c04-etc-systemd\") pod \"tuned-gbk88\" (UID: \"3d090b9a-abb1-48c3-9a43-415165ea4c04\") " pod="openshift-cluster-node-tuning-operator/tuned-gbk88" Oct 31 00:57:09.520272 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520163 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 00:57:09.520272 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520178 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.520272 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520179 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.520272 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520231 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:57:09.520272 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520235 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520313 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520366 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520465 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520492 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520528 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520565 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520578 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 00:57:09.520593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520570 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 00:57:09.520973 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520689 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 00:57:09.520973 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520727 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 00:57:09.520973 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520737 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-accelerators-collector-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.520973 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.520786 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 00:57:09.521138 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.521000 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 00:57:09.523840 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.523821 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.523971 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.523855 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.524223 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.524206 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.524332 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.524314 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74t85\" (UniqueName: \"kubernetes.io/projected/fc981b83-c624-4524-95c0-950ec6a283ee-kube-api-access-74t85\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.524513 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.524497 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fcnq\" (UniqueName: \"kubernetes.io/projected/01262e31-9355-45a2-adc6-2f7889c4fe8e-kube-api-access-2fcnq\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:57:09.525318 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525301 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c49c201-987e-4ea9-9bec-343a09101685-metrics-client-ca\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.525318 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525308 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.525440 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525321 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.525494 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525452 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8cf\" (UniqueName: \"kubernetes.io/projected/a62692fc-3f77-478d-90d5-577b92632a39-kube-api-access-2x8cf\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.525548 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525529 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.525692 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525679 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/236b6fa3-674c-4790-951e-8b91cb51deee-metrics-server-audit-profiles\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.525740 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.525725 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2b8r\" (UniqueName: \"kubernetes.io/projected/64d66bc1-aa19-496e-82c5-ec3bedaa9943-kube-api-access-h2b8r\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.528904 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.528858 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.528904 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.528899 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc981b83-c624-4524-95c0-950ec6a283ee-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.529040 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.528908 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64d66bc1-aa19-496e-82c5-ec3bedaa9943-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.529040 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.528908 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.529040 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.528933 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.529040 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.528932 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62692fc-3f77-478d-90d5-577b92632a39-metrics-client-ca\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.529938 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.529922 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.531070 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.531054 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.532477 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.531399 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq5q4\" (UniqueName: \"kubernetes.io/projected/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-api-access-kq5q4\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.570149 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.570132 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:57:09.570273 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.570257 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 00:57:09.571463 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.570909 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:57:09.571463 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.571003 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:57:09.571463 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.571293 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.571793 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.571753 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7m2\" (UniqueName: \"kubernetes.io/projected/236b6fa3-674c-4790-951e-8b91cb51deee-kube-api-access-xf7m2\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.572131 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.572109 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.572131 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.571400 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:57:09.573263 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.572351 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7mb\" (UniqueName: \"kubernetes.io/projected/73310d59-0bee-4124-ad47-b480d3544468-kube-api-access-jf7mb\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:09.573263 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.572560 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pwc\" (UniqueName: \"kubernetes.io/projected/2c49c201-987e-4ea9-9bec-343a09101685-kube-api-access-k8pwc\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.573562 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.573573 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.573839 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-367ppj0d8g0mr\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.573956 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7o3ddrn43kmg7\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.573998 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574010 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574105 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574246 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:57:09.574294 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574274 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574716 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574731 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574480 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574905 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574672 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6bhn88i2ditsv\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574991 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575094 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.574994 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kc974\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575299 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575352 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rsnwq\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575389 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575448 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575593 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575646 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575768 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575788 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575811 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575816 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.575908 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576011 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576029 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576072 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rs8xl\"" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576128 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-grpc-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.576205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576180 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ssrlm\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576027 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576378 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-r7zp6\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576427 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576449 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576451 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576449 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576497 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-client-ca-bundle\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576579 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576592 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7j8s4\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576629 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tzg24\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576634 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576662 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576688 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576685 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576715 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576765 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576819 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 00:57:09.577194 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.576978 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577301 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577325 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xxczs\"" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577367 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577368 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577411 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-n4snk\"" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577433 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.577765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.577514 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-tls\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.578131 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.578116 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.579303 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.579279 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01262e31-9355-45a2-adc6-2f7889c4fe8e-cert\") pod \"ingress-canary-gp5rq\" (UID: \"01262e31-9355-45a2-adc6-2f7889c4fe8e\") " pod="openshift-ingress-canary/ingress-canary-gp5rq" Oct 31 00:57:09.579303 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.579292 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-tls\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.579436 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.579325 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62692fc-3f77-478d-90d5-577b92632a39-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qw74k\" (UID: \"a62692fc-3f77-478d-90d5-577b92632a39\") " pod="openshift-monitoring/node-exporter-qw74k" Oct 31 00:57:09.579436 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.579346 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.579436 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.579376 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") pod \"console-7c6444d66d-l4zqt\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:09.581320 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.581296 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc981b83-c624-4524-95c0-950ec6a283ee-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-dv9w7\" (UID: \"fc981b83-c624-4524-95c0-950ec6a283ee\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-dv9w7" Oct 31 00:57:09.581387 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.581322 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-lbrgs\" (UID: \"7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-lbrgs" Oct 31 00:57:09.581387 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.581308 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64d66bc1-aa19-496e-82c5-ec3bedaa9943-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l72fc\" (UID: \"64d66bc1-aa19-496e-82c5-ec3bedaa9943\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l72fc" Oct 31 00:57:09.582471 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.582447 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.582551 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.582470 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.582551 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.582480 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.582551 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.582450 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-client-certs\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.582551 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.582458 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73310d59-0bee-4124-ad47-b480d3544468-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-x7tkk\" (UID: \"73310d59-0bee-4124-ad47-b480d3544468\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:09.582709 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.582697 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/236b6fa3-674c-4790-951e-8b91cb51deee-secret-metrics-server-tls\") pod \"metrics-server-6cf56d4544-tzfbs\" (UID: \"236b6fa3-674c-4790-951e-8b91cb51deee\") " pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:09.583051 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.583037 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.583086 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.583057 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.583841 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.583824 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") pod \"console-698867449c-xbf2l\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.583913 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.583900 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c49c201-987e-4ea9-9bec-343a09101685-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69f8cc8d77-825m2\" (UID: \"2c49c201-987e-4ea9-9bec-343a09101685\") " pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:09.583962 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.583929 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:09.586262 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586245 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586337 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586269 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdlc\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586337 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586287 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586337 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586305 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586337 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586320 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586547 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586339 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586547 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586451 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586547 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586498 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586547 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586525 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586566 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586591 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586618 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586649 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586683 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586724 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.586734 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586726 5756 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 00:57:09.587101 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586762 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587101 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586786 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587101 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586811 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587101 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586851 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587101 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.586896 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587499 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.587233 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587886 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.587844 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.587976 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.587909 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.588029 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.587967 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.588379 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.588354 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.588783 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.588757 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.589570 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.589544 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.589785 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.589760 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.589851 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.589818 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.590414 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.590314 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.590414 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.590335 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.590414 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.590314 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.590683 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.590641 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.592021 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.591996 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.592021 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.592008 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.592555 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.592529 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.592804 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.592786 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.592889 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.592819 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.593190 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.593174 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.643914 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.643885 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdlc\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc\") pod \"prometheus-k8s-0\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.675433 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.675406 5756 scope.go:122] "RemoveContainer" containerID="8bb1d1234b3099fe6534712c19fabd1cbc314ec3bacca9b03759928480580949" Oct 31 00:57:09.708151 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.708130 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:09.785267 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.785190 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:09.990497 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:09.990466 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:10.107378 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.107353 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:10.199365 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.199338 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:57:10.238084 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:10.238047 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc260129a_312f_4e9f_94ea_b476310862dd.slice/crio-faac377fa3aa466248819bb44706d7f1e872667f86665b1e3e5aef5c9cd8fa0c WatchSource:0}: Error finding container faac377fa3aa466248819bb44706d7f1e872667f86665b1e3e5aef5c9cd8fa0c: Status 404 returned error can't find the container with id faac377fa3aa466248819bb44706d7f1e872667f86665b1e3e5aef5c9cd8fa0c Oct 31 00:57:10.393843 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.393764 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" exitCode=0 Oct 31 00:57:10.394310 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.393847 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} Oct 31 00:57:10.394310 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.393940 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"faac377fa3aa466248819bb44706d7f1e872667f86665b1e3e5aef5c9cd8fa0c"} Oct 31 00:57:10.398965 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.396784 5756 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:57:10.402163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.402141 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerStarted","Data":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} Oct 31 00:57:10.405159 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.405134 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"383ca928035d2fea34ec5eb73d2ee931dd652f516c8c06d1727ccd2ad6b3a5a4"} Oct 31 00:57:10.405238 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.405183 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"a55756cffd80e97a7e26b9a7490d65a348d3195abe4e9a42d0dd659a69cc8b7d"} Oct 31 00:57:10.405288 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.405255 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" event={"ID":"2c49c201-987e-4ea9-9bec-343a09101685","Type":"ContainerStarted","Data":"93aa91289e1371d22343f6087b15863b1a8c6c7b88efe196395767f738047b60"} Oct 31 00:57:10.664215 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.664121 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6444d66d-l4zqt"] Oct 31 00:57:10.931687 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:10.931623 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wklrq" Oct 31 00:57:11.409491 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:11.409457 5756 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:57:11.610367 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:11.610334 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:11.616941 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:11.616909 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69f8cc8d77-825m2" Oct 31 00:57:12.814360 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:12.814332 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:14.074354 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.074329 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:14.428105 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.428051 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} Oct 31 00:57:14.428269 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.428114 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} Oct 31 00:57:14.428269 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.428139 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} Oct 31 00:57:14.428269 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.428163 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} Oct 31 00:57:14.428269 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.428187 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} Oct 31 00:57:14.770525 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.770495 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:14.770720 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:14.770709 5756 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 00:57:15.085924 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:15.085897 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kmmnx" Oct 31 00:57:15.438454 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:15.438360 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerStarted","Data":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} Oct 31 00:57:15.502333 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:15.502307 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:15.507174 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:15.507150 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-x7tkk" Oct 31 00:57:15.615931 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:15.615857 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.9823974030000002 podStartE2EDuration="7.615844871s" totalImagesPullingTime="3.616914114s" totalInitContainerRuntime="16.533354ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:15.615205166 +0000 UTC m=+7.422172727" watchObservedRunningTime="2026-10-31 00:57:15.615844871 +0000 UTC m=+7.422812452" Oct 31 00:57:18.705916 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:18.705856 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:19.266989 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:19.266960 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:19.790538 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:19.790509 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:19.794307 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:19.794286 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:57:19.841533 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:19.841511 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:21.576832 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:21.576800 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:24.974432 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:24.974400 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lttht" Oct 31 00:57:32.065577 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:32.065535 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 00:57:37.436798 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.436739 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c6444d66d-l4zqt" podUID="b2593841-e6ff-4bce-a8d8-a8161a7b895c" containerName="console" containerID="cri-o://ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98" gracePeriod=15 Oct 31 00:57:37.688583 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.688530 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:37.846395 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846364 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.846550 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846416 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.846550 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846457 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.846550 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846510 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.846673 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846605 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.846673 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846655 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.846779 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846697 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") pod \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\" (UID: \"b2593841-e6ff-4bce-a8d8-a8161a7b895c\") " Oct 31 00:57:37.847096 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846981 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:37.847096 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.846988 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:37.847275 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.847035 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:37.847324 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.847170 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:37.849037 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.849002 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:37.849105 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.849010 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "kube-api-access-xkvz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:37.849147 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.849089 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config" pod "b2593841-e6ff-4bce-a8d8-a8161a7b895c" (UID: "b2593841-e6ff-4bce-a8d8-a8161a7b895c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947783 5756 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947809 5756 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-oauth-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947819 5756 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-oauth-serving-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947828 5756 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2593841-e6ff-4bce-a8d8-a8161a7b895c-tmp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947837 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkvz8\" (UniqueName: \"kubernetes.io/projected/b2593841-e6ff-4bce-a8d8-a8161a7b895c-kube-api-access-xkvz8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947846 5756 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:37.947853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:37.947855 5756 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2593841-e6ff-4bce-a8d8-a8161a7b895c-console-serving-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:38.526009 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.525972 5756 generic.go:356] "Generic (PLEG): container finished" podID="b2593841-e6ff-4bce-a8d8-a8161a7b895c" containerID="ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98" exitCode=0 Oct 31 00:57:38.526447 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.526039 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6444d66d-l4zqt" Oct 31 00:57:38.526447 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.526046 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6444d66d-l4zqt" event={"ID":"b2593841-e6ff-4bce-a8d8-a8161a7b895c","Type":"ContainerDied","Data":"ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98"} Oct 31 00:57:38.526447 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.526124 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6444d66d-l4zqt" event={"ID":"b2593841-e6ff-4bce-a8d8-a8161a7b895c","Type":"ContainerDied","Data":"c94708bb314ddc80b44bf37e42072f94172b41bb3e46cf35ff6c241dab1323db"} Oct 31 00:57:38.526447 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.526144 5756 scope.go:122] "RemoveContainer" containerID="ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98" Oct 31 00:57:38.536598 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.536578 5756 scope.go:122] "RemoveContainer" containerID="ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98" Oct 31 00:57:38.536900 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:38.536876 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98\": container with ID starting with ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98 not found: ID does not exist" containerID="ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98" Oct 31 00:57:38.536976 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.536913 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98"} err="failed to get container status \"ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98\": rpc error: code = NotFound desc = could not find container \"ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98\": container with ID starting with ec02a369af3789373802df35a4a8ac6400058bcdcee286deb940705f2e9e7a98 not found: ID does not exist" Oct 31 00:57:38.692802 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.692720 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6444d66d-l4zqt"] Oct 31 00:57:38.697377 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.697354 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6444d66d-l4zqt"] Oct 31 00:57:38.711259 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.711237 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:38.714811 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:38.714792 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cf56d4544-tzfbs" Oct 31 00:57:39.269771 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:39.269745 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-blwrh" Oct 31 00:57:40.309091 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:40.309053 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2593841-e6ff-4bce-a8d8-a8161a7b895c" path="/var/lib/kubelet/pods/b2593841-e6ff-4bce-a8d8-a8161a7b895c/volumes" Oct 31 00:57:44.678215 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:44.678185 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:44.698030 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:44.697996 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:44.857130 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:44.857103 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:57:55.624252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624223 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:55.624763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624492 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="alertmanager" containerID="cri-o://12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" gracePeriod=120 Oct 31 00:57:55.624763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624561 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-web" containerID="cri-o://7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" gracePeriod=120 Oct 31 00:57:55.624763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624591 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy" containerID="cri-o://e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" gracePeriod=120 Oct 31 00:57:55.624763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624636 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="prom-label-proxy" containerID="cri-o://4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" gracePeriod=120 Oct 31 00:57:55.624763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624600 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="config-reloader" containerID="cri-o://531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" gracePeriod=120 Oct 31 00:57:55.624763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.624708 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-metric" containerID="cri-o://11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" gracePeriod=120 Oct 31 00:57:55.879767 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.879710 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:55.898255 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898230 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898397 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898274 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898397 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898345 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898397 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898382 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898554 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898420 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898554 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898449 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898554 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898476 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.898554 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898511 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898561 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898596 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898633 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898647 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898711 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.898756 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") pod \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\" (UID: \"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d\") " Oct 31 00:57:55.899293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.899204 5756 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-main-db\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:55.899769 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.899560 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:55.899912 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.899882 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:57:55.902034 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.901995 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:55.902285 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.902236 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:55.902285 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.902247 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:55.902448 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.902325 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:55.902931 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.902909 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:55.903053 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.903032 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "kube-api-access-2n7pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:57:55.903316 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.903289 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:55.903589 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.903565 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:57:55.907018 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.906849 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:55.913996 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:55.913977 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config" pod "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" (UID: "b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:57:56.000516 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000486 5756 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000516 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000513 5756 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-cluster-tls-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000516 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000524 5756 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000533 5756 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-config-out\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000542 5756 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-tls-assets\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000552 5756 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-metrics-client-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000561 5756 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000571 5756 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000580 5756 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000589 5756 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-secret-alertmanager-main-tls\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000614 5756 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-web-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.000737 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.000623 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n7pp\" (UniqueName: \"kubernetes.io/projected/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d-kube-api-access-2n7pp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:57:56.597340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597255 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" exitCode=0 Oct 31 00:57:56.597340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597285 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" exitCode=0 Oct 31 00:57:56.597340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597293 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" exitCode=0 Oct 31 00:57:56.597340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597299 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" exitCode=0 Oct 31 00:57:56.597340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597304 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" exitCode=0 Oct 31 00:57:56.597340 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597312 5756 generic.go:356] "Generic (PLEG): container finished" podID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" exitCode=0 Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597342 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597353 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597425 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597454 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597484 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597492 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597523 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597563 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} Oct 31 00:57:56.597726 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.597603 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d","Type":"ContainerDied","Data":"551b245f7d17b68ea34fc59940ed7b718c8ffcd538a3233894d3e3e3ebd3a0c8"} Oct 31 00:57:56.605230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.605212 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.612093 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.612069 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.619127 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.619103 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.626229 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.626101 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.633836 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.633820 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.644213 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.644193 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.651135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651119 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.651411 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.651390 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.651497 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651419 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} err="failed to get container status \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" Oct 31 00:57:56.651497 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651437 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.651693 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.651672 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.651736 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651701 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} err="failed to get container status \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" Oct 31 00:57:56.651736 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651720 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.651973 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.651951 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.652023 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651979 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} err="failed to get container status \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" Oct 31 00:57:56.652023 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.651991 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.652209 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.652193 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.652245 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652212 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} err="failed to get container status \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" Oct 31 00:57:56.652245 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652223 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.652437 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.652418 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.652502 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652445 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} err="failed to get container status \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" Oct 31 00:57:56.652502 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652467 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.652689 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.652672 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.652733 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652693 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} err="failed to get container status \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" Oct 31 00:57:56.652733 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652703 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.652960 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:57:56.652944 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.653015 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652966 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} err="failed to get container status \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" Oct 31 00:57:56.653015 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.652981 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.653184 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653170 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} err="failed to get container status \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" Oct 31 00:57:56.653227 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653185 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.653418 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653404 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} err="failed to get container status \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" Oct 31 00:57:56.653459 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653419 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.653641 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653623 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} err="failed to get container status \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" Oct 31 00:57:56.653712 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653644 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.653949 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653895 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} err="failed to get container status \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" Oct 31 00:57:56.653949 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.653914 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.654134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654119 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} err="failed to get container status \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" Oct 31 00:57:56.654176 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654133 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.654355 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654338 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} err="failed to get container status \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" Oct 31 00:57:56.654400 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654356 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.654559 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654540 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} err="failed to get container status \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" Oct 31 00:57:56.654625 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654561 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.654803 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654787 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} err="failed to get container status \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" Oct 31 00:57:56.654856 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.654804 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.655056 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655038 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} err="failed to get container status \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" Oct 31 00:57:56.655117 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655057 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.655279 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655261 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} err="failed to get container status \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" Oct 31 00:57:56.655345 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655280 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.655501 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655482 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} err="failed to get container status \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" Oct 31 00:57:56.655557 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655503 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.655703 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655688 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} err="failed to get container status \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" Oct 31 00:57:56.655745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655703 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.655917 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655899 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} err="failed to get container status \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" Oct 31 00:57:56.655975 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.655918 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.656147 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656130 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} err="failed to get container status \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" Oct 31 00:57:56.656197 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656149 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.656332 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656318 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} err="failed to get container status \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" Oct 31 00:57:56.656368 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656332 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.656524 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656507 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} err="failed to get container status \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" Oct 31 00:57:56.656591 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656525 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.656762 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656745 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} err="failed to get container status \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" Oct 31 00:57:56.656809 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656762 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.656983 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656966 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} err="failed to get container status \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" Oct 31 00:57:56.657028 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.656985 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.657184 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657166 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} err="failed to get container status \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" Oct 31 00:57:56.657225 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657184 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.657379 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657362 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} err="failed to get container status \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" Oct 31 00:57:56.657446 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657381 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.657625 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657605 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} err="failed to get container status \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" Oct 31 00:57:56.657666 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657626 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.657846 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657827 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} err="failed to get container status \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" Oct 31 00:57:56.657939 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.657847 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.658101 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658085 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} err="failed to get container status \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" Oct 31 00:57:56.658147 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658101 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.658293 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658278 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} err="failed to get container status \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" Oct 31 00:57:56.658334 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658293 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.658485 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658466 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} err="failed to get container status \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" Oct 31 00:57:56.658550 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658487 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.658729 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658710 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} err="failed to get container status \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" Oct 31 00:57:56.658774 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658730 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.658983 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658965 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} err="failed to get container status \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" Oct 31 00:57:56.659028 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.658984 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.659211 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659194 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} err="failed to get container status \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" Oct 31 00:57:56.659262 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659211 5756 scope.go:122] "RemoveContainer" containerID="4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73" Oct 31 00:57:56.659441 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659423 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73"} err="failed to get container status \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": rpc error: code = NotFound desc = could not find container \"4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73\": container with ID starting with 4d75b379e2db3f25ca31ea18fa343aa768ecd3ad1d9c18622bde4ea35fb3ee73 not found: ID does not exist" Oct 31 00:57:56.659490 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659441 5756 scope.go:122] "RemoveContainer" containerID="11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7" Oct 31 00:57:56.659623 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659608 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7"} err="failed to get container status \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": rpc error: code = NotFound desc = could not find container \"11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7\": container with ID starting with 11cd65085dec2a2b75158a0edeb7159992f2c5f16b51c0b8526f9e153759dff7 not found: ID does not exist" Oct 31 00:57:56.659675 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659623 5756 scope.go:122] "RemoveContainer" containerID="e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8" Oct 31 00:57:56.659829 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659810 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8"} err="failed to get container status \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": rpc error: code = NotFound desc = could not find container \"e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8\": container with ID starting with e447fb84078bee3fb3c885e1a1deba237534384447820d3fe6fb3e75a4edabc8 not found: ID does not exist" Oct 31 00:57:56.659899 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.659830 5756 scope.go:122] "RemoveContainer" containerID="7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351" Oct 31 00:57:56.660094 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660078 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351"} err="failed to get container status \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": rpc error: code = NotFound desc = could not find container \"7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351\": container with ID starting with 7ab897713951e0750cb5b38129ea825aa517a639797d78926317270018c58351 not found: ID does not exist" Oct 31 00:57:56.660094 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660093 5756 scope.go:122] "RemoveContainer" containerID="531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af" Oct 31 00:57:56.660305 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660288 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af"} err="failed to get container status \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": rpc error: code = NotFound desc = could not find container \"531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af\": container with ID starting with 531808933d1cdacdd65ef8488104f47ee458f62b7cf42215423251ed9def79af not found: ID does not exist" Oct 31 00:57:56.660305 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660304 5756 scope.go:122] "RemoveContainer" containerID="12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe" Oct 31 00:57:56.660495 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660481 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe"} err="failed to get container status \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": rpc error: code = NotFound desc = could not find container \"12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe\": container with ID starting with 12b69328c0a60d633fe7e43496ebe47f1e02decc119440ab5ce1109c38802dfe not found: ID does not exist" Oct 31 00:57:56.660541 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660495 5756 scope.go:122] "RemoveContainer" containerID="b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4" Oct 31 00:57:56.660704 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.660688 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4"} err="failed to get container status \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": rpc error: code = NotFound desc = could not find container \"b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4\": container with ID starting with b03f563121a3112bab7efd571bb7a1496f1739a5542c1254345ec5e9ca0703b4 not found: ID does not exist" Oct 31 00:57:56.769958 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.769919 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:56.819354 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.819324 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:56.854752 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.854687 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:56.855008 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.854996 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-metric" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855010 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-metric" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855027 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-web" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855032 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-web" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855040 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="init-config-reloader" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855045 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="init-config-reloader" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855054 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="config-reloader" Oct 31 00:57:56.855057 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855059 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="config-reloader" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855069 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="prom-label-proxy" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855074 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="prom-label-proxy" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855082 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855086 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855095 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b2593841-e6ff-4bce-a8d8-a8161a7b895c" containerName="console" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855099 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2593841-e6ff-4bce-a8d8-a8161a7b895c" containerName="console" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855110 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="alertmanager" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855115 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="alertmanager" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855159 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855167 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2593841-e6ff-4bce-a8d8-a8161a7b895c" containerName="console" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855174 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="alertmanager" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855181 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-metric" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855187 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="kube-rbac-proxy-web" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855192 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="config-reloader" Oct 31 00:57:56.855252 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.855198 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" containerName="prom-label-proxy" Oct 31 00:57:56.858758 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.858741 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908231 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908203 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9561ed9e-0e79-444d-936c-6177af70ab26-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908348 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908235 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908348 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908259 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9561ed9e-0e79-444d-936c-6177af70ab26-config-out\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908348 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908273 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561ed9e-0e79-444d-936c-6177af70ab26-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908495 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908381 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-config-volume\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908495 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908419 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-web-config\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908599 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908527 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908599 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908566 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9561ed9e-0e79-444d-936c-6177af70ab26-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908680 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908608 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908731 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908690 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908785 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908734 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908887 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908799 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9561ed9e-0e79-444d-936c-6177af70ab26-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.908887 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.908827 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxkg\" (UniqueName: \"kubernetes.io/projected/9561ed9e-0e79-444d-936c-6177af70ab26-kube-api-access-4kxkg\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:56.911052 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.911035 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 00:57:56.911145 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.911128 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 00:57:56.911198 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.911139 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 00:57:56.911198 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.911154 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 00:57:56.911309 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.911297 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kc974\"" Oct 31 00:57:56.911483 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.911471 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 00:57:56.912315 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.912301 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 00:57:56.912386 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.912363 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 00:57:56.913330 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.913314 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 00:57:56.962767 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.962734 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 00:57:56.963704 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:56.963681 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:57.009457 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009432 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009590 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009469 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9561ed9e-0e79-444d-936c-6177af70ab26-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009590 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009508 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009590 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009544 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009590 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009569 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009774 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009704 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9561ed9e-0e79-444d-936c-6177af70ab26-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009774 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009741 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxkg\" (UniqueName: \"kubernetes.io/projected/9561ed9e-0e79-444d-936c-6177af70ab26-kube-api-access-4kxkg\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009882 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009771 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9561ed9e-0e79-444d-936c-6177af70ab26-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009882 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009799 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009882 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009824 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9561ed9e-0e79-444d-936c-6177af70ab26-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.009882 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009844 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9561ed9e-0e79-444d-936c-6177af70ab26-config-out\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.010085 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009890 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561ed9e-0e79-444d-936c-6177af70ab26-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.010085 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009921 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-config-volume\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.010085 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.009964 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-web-config\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.011173 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.011128 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561ed9e-0e79-444d-936c-6177af70ab26-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.011456 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.011402 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9561ed9e-0e79-444d-936c-6177af70ab26-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.012889 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.012845 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9561ed9e-0e79-444d-936c-6177af70ab26-config-out\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.012889 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.012853 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.013180 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.013154 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.013611 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.013588 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.013703 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.013675 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.013949 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.013927 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9561ed9e-0e79-444d-936c-6177af70ab26-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.014075 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.014060 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-web-config\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.014128 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.014091 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-config-volume\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.014442 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.014426 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9561ed9e-0e79-444d-936c-6177af70ab26-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.068822 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.068789 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxkg\" (UniqueName: \"kubernetes.io/projected/9561ed9e-0e79-444d-936c-6177af70ab26-kube-api-access-4kxkg\") pod \"alertmanager-main-0\" (UID: \"9561ed9e-0e79-444d-936c-6177af70ab26\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.176659 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.176583 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 00:57:57.601639 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:57.601611 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 00:57:57.604551 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:57:57.604525 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9561ed9e_0e79_444d_936c_6177af70ab26.slice/crio-aa65fae85d880aab73ba6a926596e2acb1b355ba8157c9c15ee39859a68c9410 WatchSource:0}: Error finding container aa65fae85d880aab73ba6a926596e2acb1b355ba8157c9c15ee39859a68c9410: Status 404 returned error can't find the container with id aa65fae85d880aab73ba6a926596e2acb1b355ba8157c9c15ee39859a68c9410 Oct 31 00:57:58.311635 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:58.311602 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d" path="/var/lib/kubelet/pods/b6bf6ec7-e5e0-4adf-ae7b-7d7e707ee66d/volumes" Oct 31 00:57:58.608772 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:58.608684 5756 generic.go:356] "Generic (PLEG): container finished" podID="9561ed9e-0e79-444d-936c-6177af70ab26" containerID="16e4d50dca09bfc27f3bdeffa2a1a2545db96b7d33296335cfc70b8ac5135f10" exitCode=0 Oct 31 00:57:58.608949 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:58.608774 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerDied","Data":"16e4d50dca09bfc27f3bdeffa2a1a2545db96b7d33296335cfc70b8ac5135f10"} Oct 31 00:57:58.608949 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:58.608843 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"aa65fae85d880aab73ba6a926596e2acb1b355ba8157c9c15ee39859a68c9410"} Oct 31 00:57:59.616527 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.616492 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"37e578ab1a8f201117fc864df198293d65efe2c50f973aa14cc60e28f04f3844"} Oct 31 00:57:59.616919 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.616561 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"bbac05aea77f9629545183a6b7fd5ee605b91396fb921746d9afa2481468c5a6"} Oct 31 00:57:59.616919 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.616588 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"b6eedd95b3fcc98fae1153b7c068df9738b3d57a1260f0695ea77f18a817c705"} Oct 31 00:57:59.616919 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.616614 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"95c2440347ae2efa05880bef926a341656ca835d98da509866d29bcbb8234303"} Oct 31 00:57:59.616919 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.616641 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"6afd2f004c3fa229b87c0554c6571ce028cc540040daaa6e9620911e369e1a8b"} Oct 31 00:57:59.616919 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.616665 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9561ed9e-0e79-444d-936c-6177af70ab26","Type":"ContainerStarted","Data":"b55877b668e627c1cd561da4b664cdceb940407d3b4a27ffc0f6af63bf10d1b5"} Oct 31 00:57:59.740323 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:57:59.740261 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.727440494 podStartE2EDuration="3.740242544s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.80205ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:57:59.739545718 +0000 UTC m=+51.546513273" watchObservedRunningTime="2026-10-31 00:57:59.740242544 +0000 UTC m=+51.547210085" Oct 31 00:58:01.335568 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.335533 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:58:01.336134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.335830 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="prometheus" containerID="cri-o://53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" gracePeriod=600 Oct 31 00:58:01.336134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.335904 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy" containerID="cri-o://797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" gracePeriod=600 Oct 31 00:58:01.336134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.335927 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="config-reloader" containerID="cri-o://333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" gracePeriod=600 Oct 31 00:58:01.336134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.335962 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" gracePeriod=600 Oct 31 00:58:01.336134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.335907 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="thanos-sidecar" containerID="cri-o://e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" gracePeriod=600 Oct 31 00:58:01.336134 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.336090 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-web" containerID="cri-o://77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" gracePeriod=600 Oct 31 00:58:01.589201 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.589141 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:01.626278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626249 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" exitCode=0 Oct 31 00:58:01.626278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626273 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" exitCode=0 Oct 31 00:58:01.626278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626279 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" exitCode=0 Oct 31 00:58:01.626278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626285 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" exitCode=0 Oct 31 00:58:01.626278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626290 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" exitCode=0 Oct 31 00:58:01.626278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626296 5756 generic.go:356] "Generic (PLEG): container finished" podID="c260129a-312f-4e9f-94ea-b476310862dd" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" exitCode=0 Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626317 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626343 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626377 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626406 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626433 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626446 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626471 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626508 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} Oct 31 00:58:01.626708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.626547 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c260129a-312f-4e9f-94ea-b476310862dd","Type":"ContainerDied","Data":"faac377fa3aa466248819bb44706d7f1e872667f86665b1e3e5aef5c9cd8fa0c"} Oct 31 00:58:01.634202 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.634182 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.641278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.641259 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.648324 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.648298 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.655053 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655037 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.655205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655182 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655276 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655217 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655276 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655247 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655276 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655264 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655419 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655288 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655419 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655311 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655419 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655343 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655419 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655389 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655612 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655420 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655612 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655447 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655612 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655479 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.655807 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655621 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.655882 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655811 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.655882 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655814 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655878 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655924 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.655965 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656003 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656111 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656161 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656202 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656237 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656274 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs\") pod \"c260129a-312f-4e9f-94ea-b476310862dd\" (UID: \"c260129a-312f-4e9f-94ea-b476310862dd\") " Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656301 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656372 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656581 5756 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656602 5756 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656618 5756 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656634 5756 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656649 5756 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.657230 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.656834 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:01.658083 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.657609 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.658681 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.658652 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "kube-api-access-6cdlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:01.658990 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.658964 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.659153 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.659129 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:01.660059 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.660035 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:01.660630 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.660592 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.660915 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.660806 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.660915 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.660884 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.661163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.661137 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.661163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.661146 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:01.661337 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.661316 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.661603 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.661586 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.661782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.661767 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.666295 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.666276 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.671746 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.671724 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config" pod "c260129a-312f-4e9f-94ea-b476310862dd" (UID: "c260129a-312f-4e9f-94ea-b476310862dd"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:01.673747 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.673732 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.680390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.680372 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.680660 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.680641 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.680730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.680674 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} err="failed to get container status \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" Oct 31 00:58:01.680730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.680702 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.680974 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.680956 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.681010 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.680986 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} err="failed to get container status \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" Oct 31 00:58:01.681010 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681002 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.681228 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.681211 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.681296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681237 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} err="failed to get container status \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" Oct 31 00:58:01.681296 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681258 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.681476 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.681461 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.681515 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681479 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} err="failed to get container status \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" Oct 31 00:58:01.681515 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681494 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.681737 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.681720 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.681787 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681745 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} err="failed to get container status \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" Oct 31 00:58:01.681787 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.681759 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.682027 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.682009 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.682070 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682031 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} err="failed to get container status \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" Oct 31 00:58:01.682070 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682044 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.682275 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:01.682259 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.682320 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682280 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} err="failed to get container status \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" Oct 31 00:58:01.682320 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682292 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.682551 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682534 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} err="failed to get container status \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" Oct 31 00:58:01.682592 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682553 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.682779 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682762 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} err="failed to get container status \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" Oct 31 00:58:01.682841 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.682779 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.683274 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683256 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} err="failed to get container status \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" Oct 31 00:58:01.683346 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683285 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.683541 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683524 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} err="failed to get container status \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" Oct 31 00:58:01.683599 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683543 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.683763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683745 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} err="failed to get container status \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" Oct 31 00:58:01.683763 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683762 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.684002 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.683984 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} err="failed to get container status \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" Oct 31 00:58:01.684002 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684001 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.684214 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684198 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} err="failed to get container status \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" Oct 31 00:58:01.684214 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684213 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.684424 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684408 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} err="failed to get container status \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" Oct 31 00:58:01.684468 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684424 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.684625 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684609 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} err="failed to get container status \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" Oct 31 00:58:01.684676 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684625 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.684853 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684834 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} err="failed to get container status \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" Oct 31 00:58:01.684917 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.684853 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.685093 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685077 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} err="failed to get container status \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" Oct 31 00:58:01.685093 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685092 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.685307 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685288 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} err="failed to get container status \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" Oct 31 00:58:01.685360 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685306 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.685544 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685528 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} err="failed to get container status \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" Oct 31 00:58:01.685588 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685544 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.685765 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685746 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} err="failed to get container status \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" Oct 31 00:58:01.685836 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.685766 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.686032 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686013 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} err="failed to get container status \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" Oct 31 00:58:01.686032 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686032 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.686267 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686250 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} err="failed to get container status \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" Oct 31 00:58:01.686328 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686267 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.686494 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686478 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} err="failed to get container status \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" Oct 31 00:58:01.686494 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686494 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.686687 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686672 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} err="failed to get container status \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" Oct 31 00:58:01.686742 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686687 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.686908 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686857 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} err="failed to get container status \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" Oct 31 00:58:01.686946 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.686908 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.687090 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687075 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} err="failed to get container status \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" Oct 31 00:58:01.687137 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687090 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.687264 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687249 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} err="failed to get container status \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" Oct 31 00:58:01.687321 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687263 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.687485 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687469 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} err="failed to get container status \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" Oct 31 00:58:01.687524 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687484 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.687709 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687686 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} err="failed to get container status \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" Oct 31 00:58:01.687799 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687709 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.687948 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687931 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} err="failed to get container status \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" Oct 31 00:58:01.688034 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.687949 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.688184 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688166 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} err="failed to get container status \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" Oct 31 00:58:01.688232 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688185 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.688440 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688419 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} err="failed to get container status \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" Oct 31 00:58:01.688513 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688441 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.688668 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688650 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} err="failed to get container status \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" Oct 31 00:58:01.688716 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688669 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.688924 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688906 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} err="failed to get container status \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" Oct 31 00:58:01.689003 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.688925 5756 scope.go:122] "RemoveContainer" containerID="6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef" Oct 31 00:58:01.689139 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689122 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef"} err="failed to get container status \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": rpc error: code = NotFound desc = could not find container \"6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef\": container with ID starting with 6d1e0c14187b3d39d78f283a2fb56754887bfcebdc8b62658800634ba427cfef not found: ID does not exist" Oct 31 00:58:01.689184 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689140 5756 scope.go:122] "RemoveContainer" containerID="797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13" Oct 31 00:58:01.689357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689339 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13"} err="failed to get container status \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": rpc error: code = NotFound desc = could not find container \"797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13\": container with ID starting with 797044f00253ce99c32a0fcef5211dea6b4c61f9cd5819fdce3e7299aaa5aa13 not found: ID does not exist" Oct 31 00:58:01.689357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689357 5756 scope.go:122] "RemoveContainer" containerID="77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30" Oct 31 00:58:01.689559 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689540 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30"} err="failed to get container status \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": rpc error: code = NotFound desc = could not find container \"77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30\": container with ID starting with 77f2d49bd0a13c850e00dd5e6c60f62e5c1fa64f0f18bf27da5bbcb399a07d30 not found: ID does not exist" Oct 31 00:58:01.689599 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689559 5756 scope.go:122] "RemoveContainer" containerID="e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1" Oct 31 00:58:01.689780 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689764 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1"} err="failed to get container status \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": rpc error: code = NotFound desc = could not find container \"e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1\": container with ID starting with e2708a6636c5072c7c88ef5d17e2b487800a0f4964f1a384cad137b785971bd1 not found: ID does not exist" Oct 31 00:58:01.689832 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689781 5756 scope.go:122] "RemoveContainer" containerID="333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5" Oct 31 00:58:01.689995 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689979 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5"} err="failed to get container status \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": rpc error: code = NotFound desc = could not find container \"333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5\": container with ID starting with 333e68eb93209c15d31aeb027721103e33d4f84e9f27ba5969c7ea43ead034c5 not found: ID does not exist" Oct 31 00:58:01.690046 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.689996 5756 scope.go:122] "RemoveContainer" containerID="53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846" Oct 31 00:58:01.690201 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.690186 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846"} err="failed to get container status \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": rpc error: code = NotFound desc = could not find container \"53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846\": container with ID starting with 53fc1829a6b6857216e10dc8bdeef9e192b0aed09445fd569711381e87718846 not found: ID does not exist" Oct 31 00:58:01.690247 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.690202 5756 scope.go:122] "RemoveContainer" containerID="40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338" Oct 31 00:58:01.690404 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.690382 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338"} err="failed to get container status \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": rpc error: code = NotFound desc = could not find container \"40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338\": container with ID starting with 40510fda3a50bb9877fd9102f9f075631b5e0120d9e83d58c01384cfa3ea2338 not found: ID does not exist" Oct 31 00:58:01.757082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757044 5756 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-tls-assets\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757072 5756 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757084 5756 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-config-out\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757094 5756 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-db\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757103 5756 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757112 5756 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-web-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757119 5756 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-grpc-tls\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757129 5756 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-configmap-metrics-client-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757138 5756 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757147 5756 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-kube-rbac-proxy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757155 5756 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757163 5756 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c260129a-312f-4e9f-94ea-b476310862dd-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757172 5756 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-secret-metrics-client-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757180 5756 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c260129a-312f-4e9f-94ea-b476310862dd-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:01.757357 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:01.757187 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cdlc\" (UniqueName: \"kubernetes.io/projected/c260129a-312f-4e9f-94ea-b476310862dd-kube-api-access-6cdlc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:02.047157 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.047125 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:58:02.051036 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.051010 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:58:02.065071 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:02.065042 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 00:58:02.075783 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.075765 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:58:02.076117 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076102 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="config-reloader" Oct 31 00:58:02.076163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076121 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="config-reloader" Oct 31 00:58:02.076163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076136 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="init-config-reloader" Oct 31 00:58:02.076163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076142 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="init-config-reloader" Oct 31 00:58:02.076163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076148 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="thanos-sidecar" Oct 31 00:58:02.076163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076154 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="thanos-sidecar" Oct 31 00:58:02.076163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076161 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-web" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076166 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-web" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076178 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-thanos" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076183 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-thanos" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076200 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="prometheus" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076205 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="prometheus" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076215 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076219 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076267 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-web" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076279 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy-thanos" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076287 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="prometheus" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076294 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="kube-rbac-proxy" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076299 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="config-reloader" Oct 31 00:58:02.076344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.076306 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c260129a-312f-4e9f-94ea-b476310862dd" containerName="thanos-sidecar" Oct 31 00:58:02.080062 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.080044 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.133074 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.133043 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 00:58:02.133714 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.133684 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-367ppj0d8g0mr\"" Oct 31 00:58:02.133714 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.133684 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 00:58:02.134020 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.133734 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 00:58:02.134020 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.133958 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 00:58:02.134209 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134055 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 00:58:02.134209 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134063 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 00:58:02.134209 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134105 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 00:58:02.134209 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134108 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 00:58:02.134443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134320 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 00:58:02.134443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134386 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xxczs\"" Oct 31 00:58:02.134443 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134432 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 00:58:02.134779 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134746 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 00:58:02.134779 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.134754 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 00:58:02.160344 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160317 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634481e2-b617-4271-8d38-9793dc458bd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160455 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160352 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160455 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160379 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-config\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160455 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160394 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160550 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160487 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634481e2-b617-4271-8d38-9793dc458bd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160585 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160543 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160585 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160572 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160643 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160607 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160671 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160651 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160680 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160733 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160705 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160764 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160744 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160764 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160758 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160835 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160802 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160835 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160831 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hdq\" (UniqueName: \"kubernetes.io/projected/634481e2-b617-4271-8d38-9793dc458bd0-kube-api-access-d9hdq\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160953 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160911 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160985 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160951 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.160985 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160968 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.161056 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160984 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.161056 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.160999 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.188028 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.184938 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 00:58:02.188028 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.185545 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 00:58:02.262179 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262139 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262195 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262227 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262260 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262286 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262309 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262339 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262366 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262361 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262396 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262435 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hdq\" (UniqueName: \"kubernetes.io/projected/634481e2-b617-4271-8d38-9793dc458bd0-kube-api-access-d9hdq\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262464 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262511 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262532 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262557 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262580 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262616 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634481e2-b617-4271-8d38-9793dc458bd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262639 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262682 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-config\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262702 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.262745 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.262737 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634481e2-b617-4271-8d38-9793dc458bd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.266401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.265443 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.266401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.265562 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.266401 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.266125 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.267437 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.267362 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.267803 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.267778 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.268208 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.268187 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.268730 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.268702 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.269163 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.269137 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.269310 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.269288 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.269390 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.269342 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.269593 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.269522 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/634481e2-b617-4271-8d38-9793dc458bd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.269744 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.269717 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/634481e2-b617-4271-8d38-9793dc458bd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.270172 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.270144 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.270433 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.270388 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.271282 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.271255 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.272186 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.272168 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.272295 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.272168 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-config\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.273205 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.273180 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/634481e2-b617-4271-8d38-9793dc458bd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.275813 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.275775 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/634481e2-b617-4271-8d38-9793dc458bd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.299749 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.299718 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:58:02.308773 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.308739 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c260129a-312f-4e9f-94ea-b476310862dd" path="/var/lib/kubelet/pods/c260129a-312f-4e9f-94ea-b476310862dd/volumes" Oct 31 00:58:02.319898 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.319855 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hdq\" (UniqueName: \"kubernetes.io/projected/634481e2-b617-4271-8d38-9793dc458bd0-kube-api-access-d9hdq\") pod \"prometheus-k8s-0\" (UID: \"634481e2-b617-4271-8d38-9793dc458bd0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.397671 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.397640 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:02.826295 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:02.826267 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 00:58:02.829282 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:58:02.829252 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634481e2_b617_4271_8d38_9793dc458bd0.slice/crio-2da31b829e59a4358a6192fd207a512d9f86193511b4a117d7ec6e8dda57108f WatchSource:0}: Error finding container 2da31b829e59a4358a6192fd207a512d9f86193511b4a117d7ec6e8dda57108f: Status 404 returned error can't find the container with id 2da31b829e59a4358a6192fd207a512d9f86193511b4a117d7ec6e8dda57108f Oct 31 00:58:03.636732 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:03.636698 5756 generic.go:356] "Generic (PLEG): container finished" podID="634481e2-b617-4271-8d38-9793dc458bd0" containerID="35d42997928960cce8b4b1d9e49c04033926d803a00c81caac4f181084f1a90f" exitCode=0 Oct 31 00:58:03.637147 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:03.636752 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerDied","Data":"35d42997928960cce8b4b1d9e49c04033926d803a00c81caac4f181084f1a90f"} Oct 31 00:58:03.637147 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:03.636818 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"2da31b829e59a4358a6192fd207a512d9f86193511b4a117d7ec6e8dda57108f"} Oct 31 00:58:04.441004 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.440974 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-8fbc86bbb-xrxrl"] Oct 31 00:58:04.443278 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.443261 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483099 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483068 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483248 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483124 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483248 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483195 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483248 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483232 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483370 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483269 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483370 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483311 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483370 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483363 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.483507 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.483418 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w6cf\" (UniqueName: \"kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.504038 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.504000 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fbc86bbb-xrxrl"] Oct 31 00:58:04.584615 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584580 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w6cf\" (UniqueName: \"kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.584782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584636 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.584782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584692 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.584782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584715 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.584782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584734 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.584782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584767 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585085 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584788 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585085 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.584829 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585446 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.585389 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585604 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.585582 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585801 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.585779 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585898 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.585784 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.585961 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.585943 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.587429 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.587402 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.588247 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.588230 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.644142 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.644105 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"233cd005751c8b4ecb320d413b9825f6df92eba762ff40a5e11bd9e866cda958"} Oct 31 00:58:04.644608 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.644170 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"db2ba1f9c90ff7ba161c2347dc94cf478b8a7db74111931a00b9925db2ae225b"} Oct 31 00:58:04.644608 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.644205 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"7ca57b2cd95cefc5c18be66c8d7c5c21bd23da8953f54072f1741028e7d95385"} Oct 31 00:58:04.644608 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.644229 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"a2efdfec480163f7db422d94fca70fdaa861f71f26dedc4de7215feb8451c6ff"} Oct 31 00:58:04.644608 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.644253 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"b4e7d13f4a7777a06c0f4dd7c91c38e9392792366dc55446c6f579874305b21b"} Oct 31 00:58:04.644608 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.644275 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"634481e2-b617-4271-8d38-9793dc458bd0","Type":"ContainerStarted","Data":"ba428da345bf75690362a05da9ac7e5c8853aa61ec276e5f367443a2d9c68c23"} Oct 31 00:58:04.645106 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.645087 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w6cf\" (UniqueName: \"kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf\") pod \"console-8fbc86bbb-xrxrl\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.754160 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.754089 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:04.775151 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:04.775095 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.760044518 podStartE2EDuration="2.775075116s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.030598ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:04.774098037 +0000 UTC m=+56.581065583" watchObservedRunningTime="2026-10-31 00:58:04.775075116 +0000 UTC m=+56.582042659" Oct 31 00:58:05.125549 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:05.125515 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8fbc86bbb-xrxrl"] Oct 31 00:58:05.128279 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:58:05.128248 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425c69f0_6e48_4a50_a5e1_f65b9491c6ad.slice/crio-f5b3fcf4f1b1da16600844128ee8986ce330c024f892c4d83c63932fe968af09 WatchSource:0}: Error finding container f5b3fcf4f1b1da16600844128ee8986ce330c024f892c4d83c63932fe968af09: Status 404 returned error can't find the container with id f5b3fcf4f1b1da16600844128ee8986ce330c024f892c4d83c63932fe968af09 Oct 31 00:58:05.651761 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:05.651727 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fbc86bbb-xrxrl" event={"ID":"425c69f0-6e48-4a50-a5e1-f65b9491c6ad","Type":"ContainerStarted","Data":"e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582"} Oct 31 00:58:05.652298 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:05.651788 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fbc86bbb-xrxrl" event={"ID":"425c69f0-6e48-4a50-a5e1-f65b9491c6ad","Type":"ContainerStarted","Data":"f5b3fcf4f1b1da16600844128ee8986ce330c024f892c4d83c63932fe968af09"} Oct 31 00:58:05.769300 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:05.769250 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-8fbc86bbb-xrxrl" podStartSLOduration=1.769236325 podStartE2EDuration="1.769236325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:05.768538768 +0000 UTC m=+57.575506341" watchObservedRunningTime="2026-10-31 00:58:05.769236325 +0000 UTC m=+57.576203864" Oct 31 00:58:07.398029 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:07.397994 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:14.754391 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:14.754346 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:14.754906 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:14.754421 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:14.758920 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:14.758901 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:15.694731 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:15.694704 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:58:15.862139 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:15.862105 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698867449c-xbf2l"] Oct 31 00:58:26.707841 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:26.707805 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:26.723956 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:26.723931 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:26.746800 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:26.746779 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:58:32.066163 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:32.066120 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 00:58:34.495402 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.495371 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c96d49795-dlb8h"] Oct 31 00:58:34.499493 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.499478 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.569848 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.569819 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2qjz\" (UniqueName: \"kubernetes.io/projected/078342a7-6fc6-4308-aa92-2e259d681d57-kube-api-access-j2qjz\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.569848 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.569851 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078342a7-6fc6-4308-aa92-2e259d681d57-console-oauth-config\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.570054 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.569927 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-trusted-ca-bundle\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.570054 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.570002 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078342a7-6fc6-4308-aa92-2e259d681d57-console-serving-cert\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.570054 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.570037 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-service-ca\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.570144 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.570094 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-oauth-serving-cert\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.570179 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.570154 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-console-config\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.570179 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.570172 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/078342a7-6fc6-4308-aa92-2e259d681d57-tmp\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.601918 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.601891 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c96d49795-dlb8h"] Oct 31 00:58:34.670642 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670616 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-console-config\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.670812 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670648 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/078342a7-6fc6-4308-aa92-2e259d681d57-tmp\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.670812 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670673 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2qjz\" (UniqueName: \"kubernetes.io/projected/078342a7-6fc6-4308-aa92-2e259d681d57-kube-api-access-j2qjz\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.670812 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670702 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078342a7-6fc6-4308-aa92-2e259d681d57-console-oauth-config\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.670812 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670722 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-trusted-ca-bundle\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.670812 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670760 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078342a7-6fc6-4308-aa92-2e259d681d57-console-serving-cert\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.670812 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670794 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-service-ca\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.671135 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.670817 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-oauth-serving-cert\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.671182 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.671144 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/078342a7-6fc6-4308-aa92-2e259d681d57-tmp\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.671502 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.671480 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-service-ca\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.671618 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.671527 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-console-config\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.671618 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.671562 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-oauth-serving-cert\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.671992 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.671974 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/078342a7-6fc6-4308-aa92-2e259d681d57-trusted-ca-bundle\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.673925 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.673904 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/078342a7-6fc6-4308-aa92-2e259d681d57-console-serving-cert\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.674008 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.673938 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/078342a7-6fc6-4308-aa92-2e259d681d57-console-oauth-config\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.727472 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.727443 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2qjz\" (UniqueName: \"kubernetes.io/projected/078342a7-6fc6-4308-aa92-2e259d681d57-kube-api-access-j2qjz\") pod \"console-5c96d49795-dlb8h\" (UID: \"078342a7-6fc6-4308-aa92-2e259d681d57\") " pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:34.811131 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:34.811105 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:35.183713 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:35.183643 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c96d49795-dlb8h"] Oct 31 00:58:35.186581 ip-10-0-136-249 kubenswrapper[5756]: W1031 00:58:35.186548 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078342a7_6fc6_4308_aa92_2e259d681d57.slice/crio-38da58c2a41b9c8d12ae7b239fd2b1f7c93cf73117194924a4ee94148f803ebb WatchSource:0}: Error finding container 38da58c2a41b9c8d12ae7b239fd2b1f7c93cf73117194924a4ee94148f803ebb: Status 404 returned error can't find the container with id 38da58c2a41b9c8d12ae7b239fd2b1f7c93cf73117194924a4ee94148f803ebb Oct 31 00:58:35.766564 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:35.766527 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c96d49795-dlb8h" event={"ID":"078342a7-6fc6-4308-aa92-2e259d681d57","Type":"ContainerStarted","Data":"2ab4886cef6b792514635721a8e31368b5b4468990a3b926afde344ee132a595"} Oct 31 00:58:35.766962 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:35.766587 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c96d49795-dlb8h" event={"ID":"078342a7-6fc6-4308-aa92-2e259d681d57","Type":"ContainerStarted","Data":"38da58c2a41b9c8d12ae7b239fd2b1f7c93cf73117194924a4ee94148f803ebb"} Oct 31 00:58:35.882697 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:35.882137 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c96d49795-dlb8h" podStartSLOduration=1.882120265 podStartE2EDuration="1.882120265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:35.882048865 +0000 UTC m=+87.689016409" watchObservedRunningTime="2026-10-31 00:58:35.882120265 +0000 UTC m=+87.689087810" Oct 31 00:58:40.882039 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:40.881975 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-698867449c-xbf2l" podUID="f3d183d8-26d7-4e77-aa32-64e75fc435a5" containerName="console" containerID="cri-o://439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256" gracePeriod=15 Oct 31 00:58:41.144388 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.144332 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:58:41.218147 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218124 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.218260 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218156 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.218260 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218202 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.218260 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218231 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.218422 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218278 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.218478 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218420 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.218782 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218760 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:41.218967 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218811 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:41.218967 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218844 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:41.218967 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.218961 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:58:41.220396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.220371 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "kube-api-access-8cfbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:58:41.220510 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.220485 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:41.319707 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.319676 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.319883 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.319716 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") pod \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\" (UID: \"f3d183d8-26d7-4e77-aa32-64e75fc435a5\") " Oct 31 00:58:41.319968 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.319951 5756 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-oauth-serving-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.320008 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.319976 5756 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-serving-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.320008 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.319992 5756 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.320082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.320005 5756 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-trusted-ca-bundle\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.320082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.320019 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cfbt\" (UniqueName: \"kubernetes.io/projected/f3d183d8-26d7-4e77-aa32-64e75fc435a5-kube-api-access-8cfbt\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.320082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.320034 5756 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.320082 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.320057 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:58:41.321961 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.321942 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config" pod "f3d183d8-26d7-4e77-aa32-64e75fc435a5" (UID: "f3d183d8-26d7-4e77-aa32-64e75fc435a5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:58:41.421367 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.421291 5756 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3d183d8-26d7-4e77-aa32-64e75fc435a5-console-oauth-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.421367 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.421317 5756 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3d183d8-26d7-4e77-aa32-64e75fc435a5-tmp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:58:41.790487 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.790458 5756 generic.go:356] "Generic (PLEG): container finished" podID="f3d183d8-26d7-4e77-aa32-64e75fc435a5" containerID="439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256" exitCode=0 Oct 31 00:58:41.790676 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.790516 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-698867449c-xbf2l" Oct 31 00:58:41.790676 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.790544 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698867449c-xbf2l" event={"ID":"f3d183d8-26d7-4e77-aa32-64e75fc435a5","Type":"ContainerDied","Data":"439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256"} Oct 31 00:58:41.790676 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.790597 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-698867449c-xbf2l" event={"ID":"f3d183d8-26d7-4e77-aa32-64e75fc435a5","Type":"ContainerDied","Data":"01976d13abaa2d1e458fdfe9ab086bf0c63f4c80358d7421849ce75733a0a432"} Oct 31 00:58:41.790676 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.790619 5756 scope.go:122] "RemoveContainer" containerID="439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256" Oct 31 00:58:41.798952 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.798938 5756 scope.go:122] "RemoveContainer" containerID="439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256" Oct 31 00:58:41.799214 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:58:41.799194 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256\": container with ID starting with 439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256 not found: ID does not exist" containerID="439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256" Oct 31 00:58:41.799262 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.799222 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256"} err="failed to get container status \"439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256\": rpc error: code = NotFound desc = could not find container \"439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256\": container with ID starting with 439da171ce8cf6c9362645b476054b6611681e7f6f38c9e12768732c12445256 not found: ID does not exist" Oct 31 00:58:41.952218 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:41.952193 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-698867449c-xbf2l"] Oct 31 00:58:42.001708 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:42.001684 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-698867449c-xbf2l"] Oct 31 00:58:42.309001 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:42.308974 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d183d8-26d7-4e77-aa32-64e75fc435a5" path="/var/lib/kubelet/pods/f3d183d8-26d7-4e77-aa32-64e75fc435a5/volumes" Oct 31 00:58:44.812235 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:44.812211 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:44.812603 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:44.812258 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:44.816921 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:44.816899 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:45.811072 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:45.811002 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c96d49795-dlb8h" Oct 31 00:58:45.975716 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:58:45.975679 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8fbc86bbb-xrxrl"] Oct 31 00:59:02.064924 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:59:02.064889 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 00:59:10.997012 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:10.996951 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-8fbc86bbb-xrxrl" podUID="425c69f0-6e48-4a50-a5e1-f65b9491c6ad" containerName="console" containerID="cri-o://e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582" gracePeriod=15 Oct 31 00:59:11.245479 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.245456 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:59:11.385555 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385530 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp\" (UniqueName: \"kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385571 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385609 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385633 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385651 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385700 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385679 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385968 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385702 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf\" (UniqueName: \"kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385968 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385758 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert\") pod \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\" (UID: \"425c69f0-6e48-4a50-a5e1-f65b9491c6ad\") " Oct 31 00:59:11.385968 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.385831 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:59:11.386119 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.386063 5756 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-tmp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.386175 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.386130 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:11.386233 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.386171 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:11.386233 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.386196 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:11.386348 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.386281 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:59:11.387777 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.387757 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:11.387987 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.387958 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "kube-api-access-2w6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:59:11.388062 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.388043 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert" pod "425c69f0-6e48-4a50-a5e1-f65b9491c6ad" (UID: "425c69f0-6e48-4a50-a5e1-f65b9491c6ad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:59:11.486747 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486712 5756 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.486747 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486745 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w6cf\" (UniqueName: \"kubernetes.io/projected/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-kube-api-access-2w6cf\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.486980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486759 5756 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-serving-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.486980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486772 5756 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-console-oauth-config\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.486980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486785 5756 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.486980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486797 5756 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-trusted-ca-bundle\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.486980 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.486808 5756 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425c69f0-6e48-4a50-a5e1-f65b9491c6ad-oauth-serving-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 00:59:11.907314 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.907276 5756 generic.go:356] "Generic (PLEG): container finished" podID="425c69f0-6e48-4a50-a5e1-f65b9491c6ad" containerID="e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582" exitCode=0 Oct 31 00:59:11.907490 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.907325 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8fbc86bbb-xrxrl" Oct 31 00:59:11.907490 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.907363 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fbc86bbb-xrxrl" event={"ID":"425c69f0-6e48-4a50-a5e1-f65b9491c6ad","Type":"ContainerDied","Data":"e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582"} Oct 31 00:59:11.907490 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.907417 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8fbc86bbb-xrxrl" event={"ID":"425c69f0-6e48-4a50-a5e1-f65b9491c6ad","Type":"ContainerDied","Data":"f5b3fcf4f1b1da16600844128ee8986ce330c024f892c4d83c63932fe968af09"} Oct 31 00:59:11.907490 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.907442 5756 scope.go:122] "RemoveContainer" containerID="e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582" Oct 31 00:59:11.915916 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.915891 5756 scope.go:122] "RemoveContainer" containerID="e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582" Oct 31 00:59:11.916203 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:59:11.916186 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582\": container with ID starting with e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582 not found: ID does not exist" containerID="e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582" Oct 31 00:59:11.916259 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:11.916213 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582"} err="failed to get container status \"e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582\": rpc error: code = NotFound desc = could not find container \"e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582\": container with ID starting with e91593103a2d326a0d050dbfdfb1665dfba4b3efd5066383b83ea9f26a1d6582 not found: ID does not exist" Oct 31 00:59:12.078757 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:12.078728 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8fbc86bbb-xrxrl"] Oct 31 00:59:12.082078 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:12.082056 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8fbc86bbb-xrxrl"] Oct 31 00:59:12.308396 ip-10-0-136-249 kubenswrapper[5756]: I1031 00:59:12.308367 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="425c69f0-6e48-4a50-a5e1-f65b9491c6ad" path="/var/lib/kubelet/pods/425c69f0-6e48-4a50-a5e1-f65b9491c6ad/volumes" Oct 31 00:59:32.064945 ip-10-0-136-249 kubenswrapper[5756]: E1031 00:59:32.064908 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:00:02.065675 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:00:02.065637 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:00:32.064172 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:00:32.064138 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:01:02.064751 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:01:02.064716 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:01:32.064326 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:01:32.064291 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:01:46.459623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.459543 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9"] Oct 31 01:01:46.460170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.459917 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f3d183d8-26d7-4e77-aa32-64e75fc435a5" containerName="console" Oct 31 01:01:46.460170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.459933 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d183d8-26d7-4e77-aa32-64e75fc435a5" containerName="console" Oct 31 01:01:46.460170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.459975 5756 cpu_manager.go:414] "containerMap: removing container" podUID="425c69f0-6e48-4a50-a5e1-f65b9491c6ad" containerName="console" Oct 31 01:01:46.460170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.459980 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="425c69f0-6e48-4a50-a5e1-f65b9491c6ad" containerName="console" Oct 31 01:01:46.460170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.460031 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="425c69f0-6e48-4a50-a5e1-f65b9491c6ad" containerName="console" Oct 31 01:01:46.460170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.460039 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d183d8-26d7-4e77-aa32-64e75fc435a5" containerName="console" Oct 31 01:01:46.463244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.463226 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.575779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.575746 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.575984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.575788 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.575984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.575806 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/c5e5cdfe-93dd-438e-ae70-a4268581023b-kube-api-access-296f5\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.575984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.575957 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5e5cdfe-93dd-438e-ae70-a4268581023b-manifests\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.576123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.575986 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5e5cdfe-93dd-438e-ae70-a4268581023b-tmp\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.576123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.576004 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.602791 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.602759 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9"] Oct 31 01:01:46.603224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.603200 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 01:01:46.603359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.603245 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 01:01:46.603359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.603291 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wffsw\"" Oct 31 01:01:46.603359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.603312 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 01:01:46.603359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.603346 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 01:01:46.603359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.603350 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 01:01:46.677185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677157 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5e5cdfe-93dd-438e-ae70-a4268581023b-manifests\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.677341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677193 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5e5cdfe-93dd-438e-ae70-a4268581023b-tmp\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.677341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677214 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.677341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677279 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.677341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677322 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.677552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677345 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/c5e5cdfe-93dd-438e-ae70-a4268581023b-kube-api-access-296f5\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.677614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.677584 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c5e5cdfe-93dd-438e-ae70-a4268581023b-manifests\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.679614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.679586 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5e5cdfe-93dd-438e-ae70-a4268581023b-tmp\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.679901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.679853 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-metrics-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.679999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.679937 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.679999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.679937 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c5e5cdfe-93dd-438e-ae70-a4268581023b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.735500 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.735435 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296f5\" (UniqueName: \"kubernetes.io/projected/c5e5cdfe-93dd-438e-ae70-a4268581023b-kube-api-access-296f5\") pod \"opendatahub-operator-controller-manager-7fcdb566-g2dj9\" (UID: \"c5e5cdfe-93dd-438e-ae70-a4268581023b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:46.777760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:46.777732 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:47.280886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:47.280600 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9"] Oct 31 01:01:47.486315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:47.486279 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" event={"ID":"c5e5cdfe-93dd-438e-ae70-a4268581023b","Type":"ContainerStarted","Data":"c17cb03b3277cd82aba2a0ae7130f12bb2ef8f993fa40d33f452454347d184a0"} Oct 31 01:01:51.504682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:51.504594 5756 generic.go:356] "Generic (PLEG): container finished" podID="c5e5cdfe-93dd-438e-ae70-a4268581023b" containerID="71ce83f6f0a1ae7c8dcc6d49a501ad1eb3fdf97be185a902c8ecf6e802f6ed79" exitCode=0 Oct 31 01:01:51.505113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:51.504678 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" event={"ID":"c5e5cdfe-93dd-438e-ae70-a4268581023b","Type":"ContainerDied","Data":"71ce83f6f0a1ae7c8dcc6d49a501ad1eb3fdf97be185a902c8ecf6e802f6ed79"} Oct 31 01:01:52.510772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:52.510739 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" event={"ID":"c5e5cdfe-93dd-438e-ae70-a4268581023b","Type":"ContainerStarted","Data":"ad93376f6941da496e37f22c5f115382b4af26e7b613badb974bf2329aa62556"} Oct 31 01:01:52.511176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:52.510844 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:01:52.739291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:52.739196 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" podStartSLOduration=2.06207508 podStartE2EDuration="6.739179033s" totalImagesPullingTime="4.273762366s" totalInitContainerRuntime="403.341587ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:52.689609705 +0000 UTC m=+284.496577249" watchObservedRunningTime="2026-10-31 01:01:52.739179033 +0000 UTC m=+284.546146573" Oct 31 01:01:58.533921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.533890 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm"] Oct 31 01:01:58.541836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.541816 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.588000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.587972 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50b677a9-83c9-490e-bcdb-558330c160c7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wvgmm\" (UID: \"50b677a9-83c9-490e-bcdb-558330c160c7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.588180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.588043 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vpx\" (UniqueName: \"kubernetes.io/projected/50b677a9-83c9-490e-bcdb-558330c160c7-kube-api-access-55vpx\") pod \"cert-manager-webhook-5dbbbf75cc-wvgmm\" (UID: \"50b677a9-83c9-490e-bcdb-558330c160c7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.646730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.646694 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 01:01:58.646730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.646694 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 01:01:58.646957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.646756 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-kw52r\"" Oct 31 01:01:58.647755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.647733 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm"] Oct 31 01:01:58.689102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.689073 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50b677a9-83c9-490e-bcdb-558330c160c7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wvgmm\" (UID: \"50b677a9-83c9-490e-bcdb-558330c160c7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.689278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.689139 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55vpx\" (UniqueName: \"kubernetes.io/projected/50b677a9-83c9-490e-bcdb-558330c160c7-kube-api-access-55vpx\") pod \"cert-manager-webhook-5dbbbf75cc-wvgmm\" (UID: \"50b677a9-83c9-490e-bcdb-558330c160c7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.746167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.746135 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vpx\" (UniqueName: \"kubernetes.io/projected/50b677a9-83c9-490e-bcdb-558330c160c7-kube-api-access-55vpx\") pod \"cert-manager-webhook-5dbbbf75cc-wvgmm\" (UID: \"50b677a9-83c9-490e-bcdb-558330c160c7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.746488 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.746464 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50b677a9-83c9-490e-bcdb-558330c160c7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-wvgmm\" (UID: \"50b677a9-83c9-490e-bcdb-558330c160c7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:58.865518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:58.865444 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:01:59.258420 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:59.258351 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm"] Oct 31 01:01:59.261782 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:01:59.261741 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50b677a9_83c9_490e_bcdb_558330c160c7.slice/crio-21247836b48e661731ec1efcd4ae3afb810b80d3d1e664359e15bd80ebc47a63 WatchSource:0}: Error finding container 21247836b48e661731ec1efcd4ae3afb810b80d3d1e664359e15bd80ebc47a63: Status 404 returned error can't find the container with id 21247836b48e661731ec1efcd4ae3afb810b80d3d1e664359e15bd80ebc47a63 Oct 31 01:01:59.536573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:01:59.536543 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" event={"ID":"50b677a9-83c9-490e-bcdb-558330c160c7","Type":"ContainerStarted","Data":"21247836b48e661731ec1efcd4ae3afb810b80d3d1e664359e15bd80ebc47a63"} Oct 31 01:02:02.071686 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:02:02.071626 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:02:03.516649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:03.516623 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7fcdb566-g2dj9" Oct 31 01:02:04.560193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:04.560154 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" event={"ID":"50b677a9-83c9-490e-bcdb-558330c160c7","Type":"ContainerStarted","Data":"ae2f6eff15a9cb4958cfaa2e14662bf8910ee8856cb3919bdafaa0b4b3efb8b9"} Oct 31 01:02:04.560570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:04.560259 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:02:08.153464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.153406 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" podStartSLOduration=5.736492784 podStartE2EDuration="10.153389442s" totalImagesPullingTime="4.416896658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:04.681018468 +0000 UTC m=+296.487986010" watchObservedRunningTime="2026-10-31 01:02:08.153389442 +0000 UTC m=+299.960357000" Oct 31 01:02:08.154333 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.154315 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tbnvv"] Oct 31 01:02:08.156617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.156599 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.258057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.258023 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r9ghl\"" Oct 31 01:02:08.259300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.259278 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tbnvv"] Oct 31 01:02:08.276179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.276152 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53a1362f-5096-48b6-9d24-3957a917fb64-bound-sa-token\") pod \"cert-manager-597b95cb84-tbnvv\" (UID: \"53a1362f-5096-48b6-9d24-3957a917fb64\") " pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.276304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.276206 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrhc\" (UniqueName: \"kubernetes.io/projected/53a1362f-5096-48b6-9d24-3957a917fb64-kube-api-access-8xrhc\") pod \"cert-manager-597b95cb84-tbnvv\" (UID: \"53a1362f-5096-48b6-9d24-3957a917fb64\") " pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.377044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.377013 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrhc\" (UniqueName: \"kubernetes.io/projected/53a1362f-5096-48b6-9d24-3957a917fb64-kube-api-access-8xrhc\") pod \"cert-manager-597b95cb84-tbnvv\" (UID: \"53a1362f-5096-48b6-9d24-3957a917fb64\") " pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.377208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.377128 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53a1362f-5096-48b6-9d24-3957a917fb64-bound-sa-token\") pod \"cert-manager-597b95cb84-tbnvv\" (UID: \"53a1362f-5096-48b6-9d24-3957a917fb64\") " pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.433991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.433930 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/53a1362f-5096-48b6-9d24-3957a917fb64-bound-sa-token\") pod \"cert-manager-597b95cb84-tbnvv\" (UID: \"53a1362f-5096-48b6-9d24-3957a917fb64\") " pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.434105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.434061 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrhc\" (UniqueName: \"kubernetes.io/projected/53a1362f-5096-48b6-9d24-3957a917fb64-kube-api-access-8xrhc\") pod \"cert-manager-597b95cb84-tbnvv\" (UID: \"53a1362f-5096-48b6-9d24-3957a917fb64\") " pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.517558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.517525 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r9ghl\"" Oct 31 01:02:08.527469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.527443 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tbnvv" Oct 31 01:02:08.904218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:08.904187 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tbnvv"] Oct 31 01:02:08.906952 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:02:08.906922 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53a1362f_5096_48b6_9d24_3957a917fb64.slice/crio-b8194a2eea3ee620194ad3e07cf5486c8619868b2baaf9d186961e5baaf54760 WatchSource:0}: Error finding container b8194a2eea3ee620194ad3e07cf5486c8619868b2baaf9d186961e5baaf54760: Status 404 returned error can't find the container with id b8194a2eea3ee620194ad3e07cf5486c8619868b2baaf9d186961e5baaf54760 Oct 31 01:02:09.586253 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:09.586218 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tbnvv" event={"ID":"53a1362f-5096-48b6-9d24-3957a917fb64","Type":"ContainerStarted","Data":"86ab282283876bb890601ba3d1ee0703ccfe7db18626e51beb93a909afee1823"} Oct 31 01:02:09.586253 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:09.586258 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tbnvv" event={"ID":"53a1362f-5096-48b6-9d24-3957a917fb64","Type":"ContainerStarted","Data":"b8194a2eea3ee620194ad3e07cf5486c8619868b2baaf9d186961e5baaf54760"} Oct 31 01:02:09.706182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:09.706130 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tbnvv" podStartSLOduration=1.706116276 podStartE2EDuration="1.706116276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:09.705539475 +0000 UTC m=+301.512507019" watchObservedRunningTime="2026-10-31 01:02:09.706116276 +0000 UTC m=+301.513083819" Oct 31 01:02:10.565939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:10.565913 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-wvgmm" Oct 31 01:02:21.633062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.633030 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z"] Oct 31 01:02:21.635613 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.635591 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.736910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.736877 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zzlww\"" Oct 31 01:02:21.736910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.736898 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 01:02:21.736910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.736908 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 01:02:21.736910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.736927 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 01:02:21.737250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.736966 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 01:02:21.738000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.737978 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z"] Oct 31 01:02:21.802851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.802819 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17eeee37-e7d5-470d-b743-39dd06a083c2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.803027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.802948 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnh7\" (UniqueName: \"kubernetes.io/projected/17eeee37-e7d5-470d-b743-39dd06a083c2-kube-api-access-pcnh7\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.803027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.802983 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17eeee37-e7d5-470d-b743-39dd06a083c2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.904090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.904011 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnh7\" (UniqueName: \"kubernetes.io/projected/17eeee37-e7d5-470d-b743-39dd06a083c2-kube-api-access-pcnh7\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.904090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.904050 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17eeee37-e7d5-470d-b743-39dd06a083c2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.904257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.904103 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17eeee37-e7d5-470d-b743-39dd06a083c2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.904888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.904791 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/17eeee37-e7d5-470d-b743-39dd06a083c2-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.906596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.906573 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/17eeee37-e7d5-470d-b743-39dd06a083c2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:21.960851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:21.960821 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnh7\" (UniqueName: \"kubernetes.io/projected/17eeee37-e7d5-470d-b743-39dd06a083c2-kube-api-access-pcnh7\") pod \"kuadrant-console-plugin-66c4fb575d-qph5z\" (UID: \"17eeee37-e7d5-470d-b743-39dd06a083c2\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:22.245895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:22.245766 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" Oct 31 01:02:22.614450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:22.614422 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z"] Oct 31 01:02:22.617715 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:02:22.617687 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17eeee37_e7d5_470d_b743_39dd06a083c2.slice/crio-12b6d3fa71ca206fa31cd7e2e80336e79d7f51f540692d8240834f375e19b71d WatchSource:0}: Error finding container 12b6d3fa71ca206fa31cd7e2e80336e79d7f51f540692d8240834f375e19b71d: Status 404 returned error can't find the container with id 12b6d3fa71ca206fa31cd7e2e80336e79d7f51f540692d8240834f375e19b71d Oct 31 01:02:22.619099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:22.619078 5756 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:02:22.641676 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:22.641644 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" event={"ID":"17eeee37-e7d5-470d-b743-39dd06a083c2","Type":"ContainerStarted","Data":"12b6d3fa71ca206fa31cd7e2e80336e79d7f51f540692d8240834f375e19b71d"} Oct 31 01:02:28.676785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:28.676738 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" event={"ID":"17eeee37-e7d5-470d-b743-39dd06a083c2","Type":"ContainerStarted","Data":"f2cc4661c5e140c85d0df9327fece397571ec3072eef076977bdc35e74f65180"} Oct 31 01:02:28.789551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:28.789494 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qph5z" podStartSLOduration=2.257026963 podStartE2EDuration="7.78947954s" totalImagesPullingTime="5.532452577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:28.788760032 +0000 UTC m=+320.595727576" watchObservedRunningTime="2026-10-31 01:02:28.78947954 +0000 UTC m=+320.596447082" Oct 31 01:02:32.065693 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:02:32.065657 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:02:55.761130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.761092 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-w8545"] Oct 31 01:02:55.765999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.765981 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:55.869448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.869421 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:02:55.869448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.869430 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 01:02:55.869640 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.869540 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-rs95t\"" Oct 31 01:02:55.870570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.870553 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-w8545"] Oct 31 01:02:55.903713 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.903682 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87a44f6-d687-4572-8945-6399c39ce9f2-tmp\") pod \"jobset-operator-674cd59dc5-w8545\" (UID: \"d87a44f6-d687-4572-8945-6399c39ce9f2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:55.903839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:55.903729 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h69r\" (UniqueName: \"kubernetes.io/projected/d87a44f6-d687-4572-8945-6399c39ce9f2-kube-api-access-7h69r\") pod \"jobset-operator-674cd59dc5-w8545\" (UID: \"d87a44f6-d687-4572-8945-6399c39ce9f2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:56.004680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.004648 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h69r\" (UniqueName: \"kubernetes.io/projected/d87a44f6-d687-4572-8945-6399c39ce9f2-kube-api-access-7h69r\") pod \"jobset-operator-674cd59dc5-w8545\" (UID: \"d87a44f6-d687-4572-8945-6399c39ce9f2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:56.004835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.004733 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87a44f6-d687-4572-8945-6399c39ce9f2-tmp\") pod \"jobset-operator-674cd59dc5-w8545\" (UID: \"d87a44f6-d687-4572-8945-6399c39ce9f2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:56.005076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.005060 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87a44f6-d687-4572-8945-6399c39ce9f2-tmp\") pod \"jobset-operator-674cd59dc5-w8545\" (UID: \"d87a44f6-d687-4572-8945-6399c39ce9f2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:56.062603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.062573 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h69r\" (UniqueName: \"kubernetes.io/projected/d87a44f6-d687-4572-8945-6399c39ce9f2-kube-api-access-7h69r\") pod \"jobset-operator-674cd59dc5-w8545\" (UID: \"d87a44f6-d687-4572-8945-6399c39ce9f2\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:56.076135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.076113 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" Oct 31 01:02:56.442072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.441996 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-w8545"] Oct 31 01:02:56.445920 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:02:56.445896 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd87a44f6_d687_4572_8945_6399c39ce9f2.slice/crio-a636207a268352f00bb2f9229a3d5fbe63872bd4cfc763a868fdee7648d32242 WatchSource:0}: Error finding container a636207a268352f00bb2f9229a3d5fbe63872bd4cfc763a868fdee7648d32242: Status 404 returned error can't find the container with id a636207a268352f00bb2f9229a3d5fbe63872bd4cfc763a868fdee7648d32242 Oct 31 01:02:56.793533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:02:56.793493 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" event={"ID":"d87a44f6-d687-4572-8945-6399c39ce9f2","Type":"ContainerStarted","Data":"a636207a268352f00bb2f9229a3d5fbe63872bd4cfc763a868fdee7648d32242"} Oct 31 01:03:00.812578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:00.812536 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" event={"ID":"d87a44f6-d687-4572-8945-6399c39ce9f2","Type":"ContainerStarted","Data":"4a3a15095341915fa1ddb4d173b2bceb94c4dbe199682a3e90e5fe3a5e199b4a"} Oct 31 01:03:00.928281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:00.928226 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-w8545" podStartSLOduration=1.960334564 podStartE2EDuration="5.928198779s" totalImagesPullingTime="3.967864215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:00.927631225 +0000 UTC m=+352.734598849" watchObservedRunningTime="2026-10-31 01:03:00.928198779 +0000 UTC m=+352.735166323" Oct 31 01:03:02.066658 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:03:02.066613 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:03:16.281646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.281569 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx"] Oct 31 01:03:16.285062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.285042 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.300905 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.300834 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a29063c-17ab-4027-80dd-ee3eced68a39-tmp\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.301018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.300943 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srzc\" (UniqueName: \"kubernetes.io/projected/2a29063c-17ab-4027-80dd-ee3eced68a39-kube-api-access-7srzc\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.301018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.300994 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a29063c-17ab-4027-80dd-ee3eced68a39-tls-certs\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.336751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.336725 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 01:03:16.343959 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.343928 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 01:03:16.374535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.374504 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx"] Oct 31 01:03:16.401737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.401712 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a29063c-17ab-4027-80dd-ee3eced68a39-tls-certs\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.401934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.401780 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a29063c-17ab-4027-80dd-ee3eced68a39-tmp\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.401934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.401818 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srzc\" (UniqueName: \"kubernetes.io/projected/2a29063c-17ab-4027-80dd-ee3eced68a39-kube-api-access-7srzc\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.404105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.404076 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a29063c-17ab-4027-80dd-ee3eced68a39-tmp\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.417464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.417436 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 01:03:16.422572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.422553 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xj6pp\"" Oct 31 01:03:16.422656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.422594 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 01:03:16.435229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.435203 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a29063c-17ab-4027-80dd-ee3eced68a39-tls-certs\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.465455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.465430 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srzc\" (UniqueName: \"kubernetes.io/projected/2a29063c-17ab-4027-80dd-ee3eced68a39-kube-api-access-7srzc\") pod \"kube-auth-proxy-7d58c8f9d5-52lbx\" (UID: \"2a29063c-17ab-4027-80dd-ee3eced68a39\") " pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.597259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.597179 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" Oct 31 01:03:16.969333 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:16.969264 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx"] Oct 31 01:03:16.972093 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:03:16.972061 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a29063c_17ab_4027_80dd_ee3eced68a39.slice/crio-f9908838468ea226a8b162cfc631ff0ab81c21eaa724eb70e1d2c17204af2a11 WatchSource:0}: Error finding container f9908838468ea226a8b162cfc631ff0ab81c21eaa724eb70e1d2c17204af2a11: Status 404 returned error can't find the container with id f9908838468ea226a8b162cfc631ff0ab81c21eaa724eb70e1d2c17204af2a11 Oct 31 01:03:17.889741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:17.889710 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" event={"ID":"2a29063c-17ab-4027-80dd-ee3eced68a39","Type":"ContainerStarted","Data":"f9908838468ea226a8b162cfc631ff0ab81c21eaa724eb70e1d2c17204af2a11"} Oct 31 01:03:18.758973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.758930 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:18.763380 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.763351 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:18.818344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.818307 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 01:03:18.818718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.818697 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk9gx\" (UniqueName: \"kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx\") pod \"limitador-limitador-74dfcfb894-hssdr\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:18.818834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.818769 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file\") pod \"limitador-limitador-74dfcfb894-hssdr\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:18.819262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.819243 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:18.927545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.927509 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file\") pod \"limitador-limitador-74dfcfb894-hssdr\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:18.928034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.927609 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk9gx\" (UniqueName: \"kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx\") pod \"limitador-limitador-74dfcfb894-hssdr\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:18.928303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.928277 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file\") pod \"limitador-limitador-74dfcfb894-hssdr\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:18.984380 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:18.984348 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk9gx\" (UniqueName: \"kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx\") pod \"limitador-limitador-74dfcfb894-hssdr\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:19.008321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:19.008287 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:19.081110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:19.080889 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:19.463926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:19.463833 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:19.468527 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:03:19.468503 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff69595_9b0a_499e_a10c_41d9c39b2b14.slice/crio-ae6657ccc5184c49443bc82551a88f9918e1fdd4b689b9ebbf8b6bbd821d6a39 WatchSource:0}: Error finding container ae6657ccc5184c49443bc82551a88f9918e1fdd4b689b9ebbf8b6bbd821d6a39: Status 404 returned error can't find the container with id ae6657ccc5184c49443bc82551a88f9918e1fdd4b689b9ebbf8b6bbd821d6a39 Oct 31 01:03:19.902234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:19.902185 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" event={"ID":"3ff69595-9b0a-499e-a10c-41d9c39b2b14","Type":"ContainerStarted","Data":"ae6657ccc5184c49443bc82551a88f9918e1fdd4b689b9ebbf8b6bbd821d6a39"} Oct 31 01:03:20.908124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:20.908090 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" event={"ID":"2a29063c-17ab-4027-80dd-ee3eced68a39","Type":"ContainerStarted","Data":"7e40a805ef7cd37a34ddf791e938a725d1f6a33a01bb79b11362cb410568267b"} Oct 31 01:03:21.020608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.020551 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d58c8f9d5-52lbx" podStartSLOduration=1.99218335 podStartE2EDuration="5.020532989s" totalImagesPullingTime="3.028349639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:21.020140276 +0000 UTC m=+372.827107824" watchObservedRunningTime="2026-10-31 01:03:21.020532989 +0000 UTC m=+372.827500535" Oct 31 01:03:21.808769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.808709 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:03:21.812512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.812481 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:21.861236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.861199 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f35581f-76fb-4ba1-9671-38aed1743835-config-file\") pod \"limitador-limitador-7c9b87d574-wzhvt\" (UID: \"7f35581f-76fb-4ba1-9671-38aed1743835\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:21.861430 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.861331 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt27t\" (UniqueName: \"kubernetes.io/projected/7f35581f-76fb-4ba1-9671-38aed1743835-kube-api-access-pt27t\") pod \"limitador-limitador-7c9b87d574-wzhvt\" (UID: \"7f35581f-76fb-4ba1-9671-38aed1743835\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:21.896716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.896687 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:03:21.962473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.962447 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f35581f-76fb-4ba1-9671-38aed1743835-config-file\") pod \"limitador-limitador-7c9b87d574-wzhvt\" (UID: \"7f35581f-76fb-4ba1-9671-38aed1743835\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:21.962977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.962493 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt27t\" (UniqueName: \"kubernetes.io/projected/7f35581f-76fb-4ba1-9671-38aed1743835-kube-api-access-pt27t\") pod \"limitador-limitador-7c9b87d574-wzhvt\" (UID: \"7f35581f-76fb-4ba1-9671-38aed1743835\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:21.963270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:21.963247 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f35581f-76fb-4ba1-9671-38aed1743835-config-file\") pod \"limitador-limitador-7c9b87d574-wzhvt\" (UID: \"7f35581f-76fb-4ba1-9671-38aed1743835\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:22.020348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:22.020319 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt27t\" (UniqueName: \"kubernetes.io/projected/7f35581f-76fb-4ba1-9671-38aed1743835-kube-api-access-pt27t\") pod \"limitador-limitador-7c9b87d574-wzhvt\" (UID: \"7f35581f-76fb-4ba1-9671-38aed1743835\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:22.115625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:22.115542 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:03:22.126116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:22.126094 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:22.506999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:22.506805 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:03:22.509878 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:03:22.509834 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f35581f_76fb_4ba1_9671_38aed1743835.slice/crio-a74a0fbe41bfdda6424660dd17eccd1b98ad217bc96ac72115a03f9d19f992bc WatchSource:0}: Error finding container a74a0fbe41bfdda6424660dd17eccd1b98ad217bc96ac72115a03f9d19f992bc: Status 404 returned error can't find the container with id a74a0fbe41bfdda6424660dd17eccd1b98ad217bc96ac72115a03f9d19f992bc Oct 31 01:03:22.919662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:22.919632 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" event={"ID":"7f35581f-76fb-4ba1-9671-38aed1743835","Type":"ContainerStarted","Data":"a74a0fbe41bfdda6424660dd17eccd1b98ad217bc96ac72115a03f9d19f992bc"} Oct 31 01:03:23.930649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:23.930600 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" event={"ID":"7f35581f-76fb-4ba1-9671-38aed1743835","Type":"ContainerStarted","Data":"a4861a84d5059b088c1751644c8d2f5ccbf9b756c042eba60fcb5b84cb74e757"} Oct 31 01:03:23.931135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:23.930773 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:23.932264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:23.932238 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" event={"ID":"3ff69595-9b0a-499e-a10c-41d9c39b2b14","Type":"ContainerStarted","Data":"ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb"} Oct 31 01:03:23.932404 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:23.932385 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:24.044539 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:24.044476 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" podStartSLOduration=1.787294006 podStartE2EDuration="3.04445825s" totalImagesPullingTime="1.257164244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:24.043819678 +0000 UTC m=+375.850787223" watchObservedRunningTime="2026-10-31 01:03:24.04445825 +0000 UTC m=+375.851425788" Oct 31 01:03:32.065295 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:03:32.065259 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:03:34.937500 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:34.937471 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wzhvt" Oct 31 01:03:34.938145 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:34.937544 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:35.052851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.052771 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" podStartSLOduration=13.378183963 podStartE2EDuration="17.052753922s" totalImagesPullingTime="3.674569959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:24.158601405 +0000 UTC m=+375.965568947" watchObservedRunningTime="2026-10-31 01:03:35.052753922 +0000 UTC m=+386.859721464" Oct 31 01:03:35.102100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.102070 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:35.102322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.102295 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" podUID="3ff69595-9b0a-499e-a10c-41d9c39b2b14" containerName="limitador" containerID="cri-o://ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb" gracePeriod=30 Oct 31 01:03:35.642707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.642681 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:35.789353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.789328 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx\" (UniqueName: \"kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx\") pod \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " Oct 31 01:03:35.789508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.789392 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file\" (UniqueName: \"kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file\") pod \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\" (UID: \"3ff69595-9b0a-499e-a10c-41d9c39b2b14\") " Oct 31 01:03:35.789765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.789741 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file" pod "3ff69595-9b0a-499e-a10c-41d9c39b2b14" (UID: "3ff69595-9b0a-499e-a10c-41d9c39b2b14"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:03:35.791679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.791660 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx" pod "3ff69595-9b0a-499e-a10c-41d9c39b2b14" (UID: "3ff69595-9b0a-499e-a10c-41d9c39b2b14"). InnerVolumeSpecName "kube-api-access-zk9gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:03:35.891086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.891049 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk9gx\" (UniqueName: \"kubernetes.io/projected/3ff69595-9b0a-499e-a10c-41d9c39b2b14-kube-api-access-zk9gx\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:35.891086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.891080 5756 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3ff69595-9b0a-499e-a10c-41d9c39b2b14-config-file\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:03:35.985435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.985405 5756 generic.go:356] "Generic (PLEG): container finished" podID="3ff69595-9b0a-499e-a10c-41d9c39b2b14" containerID="ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb" exitCode=0 Oct 31 01:03:35.985893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.985442 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" event={"ID":"3ff69595-9b0a-499e-a10c-41d9c39b2b14","Type":"ContainerDied","Data":"ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb"} Oct 31 01:03:35.985893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.985455 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" Oct 31 01:03:35.985893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.985485 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hssdr" event={"ID":"3ff69595-9b0a-499e-a10c-41d9c39b2b14","Type":"ContainerDied","Data":"ae6657ccc5184c49443bc82551a88f9918e1fdd4b689b9ebbf8b6bbd821d6a39"} Oct 31 01:03:35.985893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.985505 5756 scope.go:122] "RemoveContainer" containerID="ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb" Oct 31 01:03:35.994911 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.994891 5756 scope.go:122] "RemoveContainer" containerID="ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb" Oct 31 01:03:35.995189 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:03:35.995169 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb\": container with ID starting with ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb not found: ID does not exist" containerID="ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb" Oct 31 01:03:35.995237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:35.995198 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb"} err="failed to get container status \"ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb\": rpc error: code = NotFound desc = could not find container \"ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb\": container with ID starting with ee4d3561df1b967a286b980be9f9011a06492f2a11ea5c232abbb91bad9a3efb not found: ID does not exist" Oct 31 01:03:36.146470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:36.146385 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:36.196161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:36.196126 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hssdr"] Oct 31 01:03:36.310228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:36.310194 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff69595-9b0a-499e-a10c-41d9c39b2b14" path="/var/lib/kubelet/pods/3ff69595-9b0a-499e-a10c-41d9c39b2b14/volumes" Oct 31 01:03:52.119119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.119089 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2"] Oct 31 01:03:52.119477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.119461 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3ff69595-9b0a-499e-a10c-41d9c39b2b14" containerName="limitador" Oct 31 01:03:52.119477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.119471 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff69595-9b0a-499e-a10c-41d9c39b2b14" containerName="limitador" Oct 31 01:03:52.119560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.119550 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff69595-9b0a-499e-a10c-41d9c39b2b14" containerName="limitador" Oct 31 01:03:52.123353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.123339 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.229304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.229271 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vlrh9\"" Oct 31 01:03:52.229304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.229290 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:03:52.229304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.229277 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:03:52.229571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.229381 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 01:03:52.230533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.230513 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2"] Oct 31 01:03:52.246487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.246464 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgq6b\" (UniqueName: \"kubernetes.io/projected/774af864-df87-4b5c-924d-4884df9e35da-kube-api-access-hgq6b\") pod \"postgres-7cdbf6dc44-kxqm2\" (UID: \"774af864-df87-4b5c-924d-4884df9e35da\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.246607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.246508 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/774af864-df87-4b5c-924d-4884df9e35da-data\") pod \"postgres-7cdbf6dc44-kxqm2\" (UID: \"774af864-df87-4b5c-924d-4884df9e35da\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.347609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.347583 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgq6b\" (UniqueName: \"kubernetes.io/projected/774af864-df87-4b5c-924d-4884df9e35da-kube-api-access-hgq6b\") pod \"postgres-7cdbf6dc44-kxqm2\" (UID: \"774af864-df87-4b5c-924d-4884df9e35da\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.347798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.347628 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/774af864-df87-4b5c-924d-4884df9e35da-data\") pod \"postgres-7cdbf6dc44-kxqm2\" (UID: \"774af864-df87-4b5c-924d-4884df9e35da\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.347978 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.347960 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/774af864-df87-4b5c-924d-4884df9e35da-data\") pod \"postgres-7cdbf6dc44-kxqm2\" (UID: \"774af864-df87-4b5c-924d-4884df9e35da\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.407728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.407658 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgq6b\" (UniqueName: \"kubernetes.io/projected/774af864-df87-4b5c-924d-4884df9e35da-kube-api-access-hgq6b\") pod \"postgres-7cdbf6dc44-kxqm2\" (UID: \"774af864-df87-4b5c-924d-4884df9e35da\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.435263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.435241 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:03:52.805988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:52.805940 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2"] Oct 31 01:03:52.808560 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:03:52.808529 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774af864_df87_4b5c_924d_4884df9e35da.slice/crio-da65b6aeb7980b5d60953a80d3e09a296d2d7aeb99285c1ec41ff0a0cd52724c WatchSource:0}: Error finding container da65b6aeb7980b5d60953a80d3e09a296d2d7aeb99285c1ec41ff0a0cd52724c: Status 404 returned error can't find the container with id da65b6aeb7980b5d60953a80d3e09a296d2d7aeb99285c1ec41ff0a0cd52724c Oct 31 01:03:53.061615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:03:53.061532 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" event={"ID":"774af864-df87-4b5c-924d-4884df9e35da","Type":"ContainerStarted","Data":"da65b6aeb7980b5d60953a80d3e09a296d2d7aeb99285c1ec41ff0a0cd52724c"} Oct 31 01:04:00.103152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:00.103120 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" event={"ID":"774af864-df87-4b5c-924d-4884df9e35da","Type":"ContainerStarted","Data":"508c24620609394574e8a9beaa118f134f003854071ec5110e5b5861b115f893"} Oct 31 01:04:00.103562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:00.103184 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:04:00.217826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:00.217776 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" podStartSLOduration=1.595401396 podStartE2EDuration="8.217760999s" totalImagesPullingTime="6.622359603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:00.217215636 +0000 UTC m=+412.024183198" watchObservedRunningTime="2026-10-31 01:04:00.217760999 +0000 UTC m=+412.024728535" Oct 31 01:04:02.065058 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:04:02.065022 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:04:06.136516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:06.136492 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kxqm2" Oct 31 01:04:32.066181 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:04:32.066145 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:04:34.080004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.079965 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7668b86557-dm49w"] Oct 31 01:04:34.096926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.096900 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.140646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.140611 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.140793 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.140739 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.140793 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.140768 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stld6\" (UniqueName: \"kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.201708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.201679 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-5jkn2\"" Oct 31 01:04:34.201708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.201679 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 01:04:34.201945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.201679 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 01:04:34.201945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.201730 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 01:04:34.201945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.201747 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 01:04:34.201945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.201774 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 01:04:34.204147 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.204128 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7668b86557-dm49w"] Oct 31 01:04:34.241450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.241418 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.241608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.241531 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.241608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.241555 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stld6\" (UniqueName: \"kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.243968 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.243944 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.244083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.243979 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.300809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.300774 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stld6\" (UniqueName: \"kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6\") pod \"maas-controller-7668b86557-dm49w\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.408050 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.407956 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:34.817097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.817070 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6887cffc9f-lvrkf"] Oct 31 01:04:34.819890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.819854 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.849150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.849129 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3278b6-5de5-44bd-b388-a869169d754b-metrics-certs\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.849291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.849164 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2tk\" (UniqueName: \"kubernetes.io/projected/be3278b6-5de5-44bd-b388-a869169d754b-kube-api-access-zb2tk\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.849291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.849203 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be3278b6-5de5-44bd-b388-a869169d754b-webhook-cert\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.850968 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.850946 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7668b86557-dm49w"] Oct 31 01:04:34.853774 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:04:34.853751 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bec72ab_d3bd_4a6d_b1e9_c562c2d99195.slice/crio-19c413c75b38ab91afc40adc5f94c43e25bd13a7021a0d27d147e24f25c51dc2 WatchSource:0}: Error finding container 19c413c75b38ab91afc40adc5f94c43e25bd13a7021a0d27d147e24f25c51dc2: Status 404 returned error can't find the container with id 19c413c75b38ab91afc40adc5f94c43e25bd13a7021a0d27d147e24f25c51dc2 Oct 31 01:04:34.918894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.918836 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6887cffc9f-lvrkf"] Oct 31 01:04:34.950268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.950237 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be3278b6-5de5-44bd-b388-a869169d754b-webhook-cert\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.950408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.950334 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3278b6-5de5-44bd-b388-a869169d754b-metrics-certs\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.950408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.950361 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2tk\" (UniqueName: \"kubernetes.io/projected/be3278b6-5de5-44bd-b388-a869169d754b-kube-api-access-zb2tk\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.952901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.952877 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be3278b6-5de5-44bd-b388-a869169d754b-metrics-certs\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:34.953012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:34.952921 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/be3278b6-5de5-44bd-b388-a869169d754b-webhook-cert\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:35.008536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:35.008502 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2tk\" (UniqueName: \"kubernetes.io/projected/be3278b6-5de5-44bd-b388-a869169d754b-kube-api-access-zb2tk\") pod \"maas-controller-6887cffc9f-lvrkf\" (UID: \"be3278b6-5de5-44bd-b388-a869169d754b\") " pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:35.132424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:35.132352 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:35.255441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:35.255395 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7668b86557-dm49w" event={"ID":"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195","Type":"ContainerStarted","Data":"19c413c75b38ab91afc40adc5f94c43e25bd13a7021a0d27d147e24f25c51dc2"} Oct 31 01:04:35.515347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:35.515011 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6887cffc9f-lvrkf"] Oct 31 01:04:35.517654 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:04:35.517625 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3278b6_5de5_44bd_b388_a869169d754b.slice/crio-0646e8a778a8d3ef46cda4da3b5250c0475ec71d7d42a3550cb8c4e751390c45 WatchSource:0}: Error finding container 0646e8a778a8d3ef46cda4da3b5250c0475ec71d7d42a3550cb8c4e751390c45: Status 404 returned error can't find the container with id 0646e8a778a8d3ef46cda4da3b5250c0475ec71d7d42a3550cb8c4e751390c45 Oct 31 01:04:36.263601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:36.263564 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" event={"ID":"be3278b6-5de5-44bd-b388-a869169d754b","Type":"ContainerStarted","Data":"0646e8a778a8d3ef46cda4da3b5250c0475ec71d7d42a3550cb8c4e751390c45"} Oct 31 01:04:41.391270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.391231 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-skln4"] Oct 31 01:04:41.395509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.395488 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.505636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.505580 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 01:04:41.505832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.505674 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6wjq4\"" Oct 31 01:04:41.507278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.507247 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-skln4"] Oct 31 01:04:41.517270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.517243 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/269646f7-4e2b-41ba-9500-73d1f9874888-cert\") pod \"kserve-controller-manager-776cc66d47-skln4\" (UID: \"269646f7-4e2b-41ba-9500-73d1f9874888\") " pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.517400 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.517301 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79868\" (UniqueName: \"kubernetes.io/projected/269646f7-4e2b-41ba-9500-73d1f9874888-kube-api-access-79868\") pod \"kserve-controller-manager-776cc66d47-skln4\" (UID: \"269646f7-4e2b-41ba-9500-73d1f9874888\") " pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.557887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.557835 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q"] Oct 31 01:04:41.560598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.560577 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.618358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.618325 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrvk\" (UniqueName: \"kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk\") pod \"llmisvc-controller-manager-869d5dcc5d-t2f7q\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.618552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.618412 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/269646f7-4e2b-41ba-9500-73d1f9874888-cert\") pod \"kserve-controller-manager-776cc66d47-skln4\" (UID: \"269646f7-4e2b-41ba-9500-73d1f9874888\") " pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.618552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.618438 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t2f7q\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.618552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.618478 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79868\" (UniqueName: \"kubernetes.io/projected/269646f7-4e2b-41ba-9500-73d1f9874888-kube-api-access-79868\") pod \"kserve-controller-manager-776cc66d47-skln4\" (UID: \"269646f7-4e2b-41ba-9500-73d1f9874888\") " pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.621066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.621042 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q"] Oct 31 01:04:41.622314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.622293 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2dz4p\"" Oct 31 01:04:41.622447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.622354 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 01:04:41.624362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.624337 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/269646f7-4e2b-41ba-9500-73d1f9874888-cert\") pod \"kserve-controller-manager-776cc66d47-skln4\" (UID: \"269646f7-4e2b-41ba-9500-73d1f9874888\") " pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.654225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.654143 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-km9nx"] Oct 31 01:04:41.656898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.656873 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.682596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.682375 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79868\" (UniqueName: \"kubernetes.io/projected/269646f7-4e2b-41ba-9500-73d1f9874888-kube-api-access-79868\") pod \"kserve-controller-manager-776cc66d47-skln4\" (UID: \"269646f7-4e2b-41ba-9500-73d1f9874888\") " pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.711078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.711055 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:41.719626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.719602 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svf9t\" (UniqueName: \"kubernetes.io/projected/b81442f8-6450-4b10-8689-f6116b037c66-kube-api-access-svf9t\") pod \"model-serving-api-75bd89fdfc-km9nx\" (UID: \"b81442f8-6450-4b10-8689-f6116b037c66\") " pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.719711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.719636 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrvk\" (UniqueName: \"kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk\") pod \"llmisvc-controller-manager-869d5dcc5d-t2f7q\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.719745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.719711 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t2f7q\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.719783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.719767 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81442f8-6450-4b10-8689-f6116b037c66-tls-certs\") pod \"model-serving-api-75bd89fdfc-km9nx\" (UID: \"b81442f8-6450-4b10-8689-f6116b037c66\") " pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.722142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.722123 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-t2f7q\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.726691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.726669 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 01:04:41.726785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.726711 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-57b9m\"" Oct 31 01:04:41.730227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.730194 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-km9nx"] Oct 31 01:04:41.797121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.786638 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrvk\" (UniqueName: \"kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk\") pod \"llmisvc-controller-manager-869d5dcc5d-t2f7q\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.797121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.788135 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-2j77z"] Oct 31 01:04:41.797121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.791241 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:41.821218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.821191 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svf9t\" (UniqueName: \"kubernetes.io/projected/b81442f8-6450-4b10-8689-f6116b037c66-kube-api-access-svf9t\") pod \"model-serving-api-75bd89fdfc-km9nx\" (UID: \"b81442f8-6450-4b10-8689-f6116b037c66\") " pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.821366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.821253 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81442f8-6450-4b10-8689-f6116b037c66-tls-certs\") pod \"model-serving-api-75bd89fdfc-km9nx\" (UID: \"b81442f8-6450-4b10-8689-f6116b037c66\") " pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.823736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.823714 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b81442f8-6450-4b10-8689-f6116b037c66-tls-certs\") pod \"model-serving-api-75bd89fdfc-km9nx\" (UID: \"b81442f8-6450-4b10-8689-f6116b037c66\") " pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.849456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.849401 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2j77z"] Oct 31 01:04:41.849632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.849467 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-hsv8w\"" Oct 31 01:04:41.849632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.849541 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 31 01:04:41.849632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.849623 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 31 01:04:41.874562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.874535 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:41.880404 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.880382 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svf9t\" (UniqueName: \"kubernetes.io/projected/b81442f8-6450-4b10-8689-f6116b037c66-kube-api-access-svf9t\") pod \"model-serving-api-75bd89fdfc-km9nx\" (UID: \"b81442f8-6450-4b10-8689-f6116b037c66\") " pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:41.922465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.922437 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81369cfa-2441-4b29-8a48-bf452d96b522-cert\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:41.922591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.922512 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81369cfa-2441-4b29-8a48-bf452d96b522-metrics-certs\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:41.922591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.922554 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz4zh\" (UniqueName: \"kubernetes.io/projected/81369cfa-2441-4b29-8a48-bf452d96b522-kube-api-access-tz4zh\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:41.969079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:41.969048 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:42.023448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.023416 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81369cfa-2441-4b29-8a48-bf452d96b522-metrics-certs\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.023557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.023477 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz4zh\" (UniqueName: \"kubernetes.io/projected/81369cfa-2441-4b29-8a48-bf452d96b522-kube-api-access-tz4zh\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.023622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.023566 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81369cfa-2441-4b29-8a48-bf452d96b522-cert\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.027724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.027699 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81369cfa-2441-4b29-8a48-bf452d96b522-cert\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.028068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.028048 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81369cfa-2441-4b29-8a48-bf452d96b522-metrics-certs\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.094294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.094263 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz4zh\" (UniqueName: \"kubernetes.io/projected/81369cfa-2441-4b29-8a48-bf452d96b522-kube-api-access-tz4zh\") pod \"odh-model-controller-769875db56-2j77z\" (UID: \"81369cfa-2441-4b29-8a48-bf452d96b522\") " pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.101486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.101465 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-skln4"] Oct 31 01:04:42.103924 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:04:42.103874 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269646f7_4e2b_41ba_9500_73d1f9874888.slice/crio-9560aeafa8bde100b95e24a98f41bc605ec76391329bf38f8f97142ab956c804 WatchSource:0}: Error finding container 9560aeafa8bde100b95e24a98f41bc605ec76391329bf38f8f97142ab956c804: Status 404 returned error can't find the container with id 9560aeafa8bde100b95e24a98f41bc605ec76391329bf38f8f97142ab956c804 Oct 31 01:04:42.110449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.110425 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:42.245052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.244968 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q"] Oct 31 01:04:42.247182 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:04:42.247154 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podab4229c9_2845_4477_9773_93bbb475ff13.slice/crio-a47468b7aac16ba201c07715855c2b340fbf206e0a571166f7600a7cf75b28c7 WatchSource:0}: Error finding container a47468b7aac16ba201c07715855c2b340fbf206e0a571166f7600a7cf75b28c7: Status 404 returned error can't find the container with id a47468b7aac16ba201c07715855c2b340fbf206e0a571166f7600a7cf75b28c7 Oct 31 01:04:42.295938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.295907 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" event={"ID":"be3278b6-5de5-44bd-b388-a869169d754b","Type":"ContainerStarted","Data":"f3f78e277c8c12a05912c8a9bc90800adb98a360345a85adb53f89076fee1348"} Oct 31 01:04:42.296133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.296021 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:42.297407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.297376 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7668b86557-dm49w" event={"ID":"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195","Type":"ContainerStarted","Data":"329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832"} Oct 31 01:04:42.298053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.298025 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:42.298667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.298645 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" event={"ID":"ab4229c9-2845-4477-9773-93bbb475ff13","Type":"ContainerStarted","Data":"a47468b7aac16ba201c07715855c2b340fbf206e0a571166f7600a7cf75b28c7"} Oct 31 01:04:42.300072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.300052 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" event={"ID":"269646f7-4e2b-41ba-9500-73d1f9874888","Type":"ContainerStarted","Data":"9560aeafa8bde100b95e24a98f41bc605ec76391329bf38f8f97142ab956c804"} Oct 31 01:04:42.340355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.340226 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-km9nx"] Oct 31 01:04:42.343099 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:04:42.343071 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81442f8_6450_4b10_8689_f6116b037c66.slice/crio-def646f46350627852c3cf945ac971e3886d9a93d62f73405abfdcedffe53e71 WatchSource:0}: Error finding container def646f46350627852c3cf945ac971e3886d9a93d62f73405abfdcedffe53e71: Status 404 returned error can't find the container with id def646f46350627852c3cf945ac971e3886d9a93d62f73405abfdcedffe53e71 Oct 31 01:04:42.415089 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.415030 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" podStartSLOduration=2.250755787 podStartE2EDuration="8.415013168s" totalImagesPullingTime="6.164257381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:42.414088441 +0000 UTC m=+454.221055984" watchObservedRunningTime="2026-10-31 01:04:42.415013168 +0000 UTC m=+454.221980713" Oct 31 01:04:42.483458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.483430 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-2j77z"] Oct 31 01:04:42.486870 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:04:42.486831 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81369cfa_2441_4b29_8a48_bf452d96b522.slice/crio-6d28fe992e92f7848fc1fd5b30dcc1347a279e4e46d2307f71e2807e889ef73c WatchSource:0}: Error finding container 6d28fe992e92f7848fc1fd5b30dcc1347a279e4e46d2307f71e2807e889ef73c: Status 404 returned error can't find the container with id 6d28fe992e92f7848fc1fd5b30dcc1347a279e4e46d2307f71e2807e889ef73c Oct 31 01:04:42.535363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:42.535312 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7668b86557-dm49w" podStartSLOduration=1.697279872 podStartE2EDuration="8.535297224s" totalImagesPullingTime="6.838017352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:42.532610131 +0000 UTC m=+454.339577673" watchObservedRunningTime="2026-10-31 01:04:42.535297224 +0000 UTC m=+454.342264767" Oct 31 01:04:43.308743 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:43.308701 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2j77z" event={"ID":"81369cfa-2441-4b29-8a48-bf452d96b522","Type":"ContainerStarted","Data":"6d28fe992e92f7848fc1fd5b30dcc1347a279e4e46d2307f71e2807e889ef73c"} Oct 31 01:04:43.310282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:43.310248 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" event={"ID":"b81442f8-6450-4b10-8689-f6116b037c66","Type":"ContainerStarted","Data":"def646f46350627852c3cf945ac971e3886d9a93d62f73405abfdcedffe53e71"} Oct 31 01:04:46.329850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.329809 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" event={"ID":"ab4229c9-2845-4477-9773-93bbb475ff13","Type":"ContainerStarted","Data":"6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155"} Oct 31 01:04:46.330347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.329963 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:04:46.332113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.332082 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" event={"ID":"269646f7-4e2b-41ba-9500-73d1f9874888","Type":"ContainerStarted","Data":"745200e0a1242cbd2033536052cf755b84745b6e1c0e57dfa6cef5af3ac87630"} Oct 31 01:04:46.332648 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.332627 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:04:46.334684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.334661 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" event={"ID":"b81442f8-6450-4b10-8689-f6116b037c66","Type":"ContainerStarted","Data":"d137f2aa7ca1ed344d2bc13c4e568ca8e5a1b5388570ba5ac040e77b5ad53dba"} Oct 31 01:04:46.334803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.334764 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:46.444898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.444726 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" podStartSLOduration=1.920881152 podStartE2EDuration="5.444711247s" totalImagesPullingTime="3.523830095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:46.444203258 +0000 UTC m=+458.251170805" watchObservedRunningTime="2026-10-31 01:04:46.444711247 +0000 UTC m=+458.251678792" Oct 31 01:04:46.559224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.558999 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" podStartSLOduration=2.301549781 podStartE2EDuration="5.558983028s" totalImagesPullingTime="3.257433247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:46.557781668 +0000 UTC m=+458.364749212" watchObservedRunningTime="2026-10-31 01:04:46.558983028 +0000 UTC m=+458.365950572" Oct 31 01:04:46.671716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:46.671591 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" podStartSLOduration=2.494483962 podStartE2EDuration="5.671571325s" totalImagesPullingTime="3.177087363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:46.671344425 +0000 UTC m=+458.478311969" watchObservedRunningTime="2026-10-31 01:04:46.671571325 +0000 UTC m=+458.478538877" Oct 31 01:04:47.340532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:47.340494 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-2j77z" event={"ID":"81369cfa-2441-4b29-8a48-bf452d96b522","Type":"ContainerStarted","Data":"275b479a370ef36100147367bad60ef46a00ae52109f5d6aea799461139551e1"} Oct 31 01:04:47.341044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:47.340703 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:04:47.465889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:47.464537 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-2j77z" podStartSLOduration=2.506760435 podStartE2EDuration="6.464518963s" totalImagesPullingTime="3.957758528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:47.461778323 +0000 UTC m=+459.268745868" watchObservedRunningTime="2026-10-31 01:04:47.464518963 +0000 UTC m=+459.271486506" Oct 31 01:04:53.313410 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.313381 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6887cffc9f-lvrkf" Oct 31 01:04:53.479506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.479458 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7668b86557-dm49w"] Oct 31 01:04:53.479810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.479738 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7668b86557-dm49w" podUID="3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" containerName="manager" containerID="cri-o://329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832" gracePeriod=30 Oct 31 01:04:53.484687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.484655 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:53.731707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.731681 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:53.861318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.861234 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6\" (UniqueName: \"kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6\") pod \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " Oct 31 01:04:53.861465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.861398 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert\") pod \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " Oct 31 01:04:53.861465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.861453 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs\") pod \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\" (UID: \"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195\") " Oct 31 01:04:53.864379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.864345 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert" pod "3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" (UID: "3bec72ab-d3bd-4a6d-b1e9-c562c2d99195"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:53.864966 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.864944 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs" pod "3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" (UID: "3bec72ab-d3bd-4a6d-b1e9-c562c2d99195"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:04:53.865311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.865293 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6" pod "3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" (UID: "3bec72ab-d3bd-4a6d-b1e9-c562c2d99195"). InnerVolumeSpecName "kube-api-access-stld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:04:53.962548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.962520 5756 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-webhook-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:04:53.962548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.962545 5756 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-metrics-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:04:53.962718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:53.962556 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stld6\" (UniqueName: \"kubernetes.io/projected/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195-kube-api-access-stld6\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:04:54.375794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.375761 5756 generic.go:356] "Generic (PLEG): container finished" podID="3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" containerID="329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832" exitCode=0 Oct 31 01:04:54.376269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.375809 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7668b86557-dm49w" Oct 31 01:04:54.376269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.375846 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7668b86557-dm49w" event={"ID":"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195","Type":"ContainerDied","Data":"329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832"} Oct 31 01:04:54.376269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.375919 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7668b86557-dm49w" event={"ID":"3bec72ab-d3bd-4a6d-b1e9-c562c2d99195","Type":"ContainerDied","Data":"19c413c75b38ab91afc40adc5f94c43e25bd13a7021a0d27d147e24f25c51dc2"} Oct 31 01:04:54.376269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.375954 5756 scope.go:122] "RemoveContainer" containerID="329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832" Oct 31 01:04:54.384388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.384366 5756 scope.go:122] "RemoveContainer" containerID="329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832" Oct 31 01:04:54.384672 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:04:54.384651 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832\": container with ID starting with 329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832 not found: ID does not exist" containerID="329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832" Oct 31 01:04:54.384725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.384685 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832"} err="failed to get container status \"329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832\": rpc error: code = NotFound desc = could not find container \"329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832\": container with ID starting with 329eb648db6e88a4305027e3390d0c3f952c3c58c334581bb20d3e4406edf832 not found: ID does not exist" Oct 31 01:04:54.537599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.537570 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7668b86557-dm49w"] Oct 31 01:04:54.586979 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:54.586894 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7668b86557-dm49w"] Oct 31 01:04:56.309925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:56.309885 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" path="/var/lib/kubelet/pods/3bec72ab-d3bd-4a6d-b1e9-c562c2d99195/volumes" Oct 31 01:04:57.345547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:57.345510 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-km9nx" Oct 31 01:04:58.347264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:04:58.347234 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-2j77z" Oct 31 01:05:02.066212 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:05:02.066177 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:05:17.343232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:05:17.343199 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:05:18.350314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:05:18.350286 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-skln4" Oct 31 01:05:24.767941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:05:24.767908 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:05:32.067715 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:05:32.067679 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:06:01.776694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.776647 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p"] Oct 31 01:06:01.777170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.777098 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" containerName="manager" Oct 31 01:06:01.777170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.777112 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" containerName="manager" Oct 31 01:06:01.777239 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.777204 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bec72ab-d3bd-4a6d-b1e9-c562c2d99195" containerName="manager" Oct 31 01:06:01.779387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.779367 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:01.836467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.836431 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p"] Oct 31 01:06:01.896550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.896513 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flddd\" (UniqueName: \"kubernetes.io/projected/b793557f-8c8e-4def-85b6-7aa0b2c5e9ee-kube-api-access-flddd\") pod \"llmisvc-controller-manager-6466c554d8-bgs5p\" (UID: \"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:01.896741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.896588 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b793557f-8c8e-4def-85b6-7aa0b2c5e9ee-cert\") pod \"llmisvc-controller-manager-6466c554d8-bgs5p\" (UID: \"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:01.997845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.997813 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b793557f-8c8e-4def-85b6-7aa0b2c5e9ee-cert\") pod \"llmisvc-controller-manager-6466c554d8-bgs5p\" (UID: \"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:01.998035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:01.997922 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flddd\" (UniqueName: \"kubernetes.io/projected/b793557f-8c8e-4def-85b6-7aa0b2c5e9ee-kube-api-access-flddd\") pod \"llmisvc-controller-manager-6466c554d8-bgs5p\" (UID: \"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:02.000450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:02.000424 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b793557f-8c8e-4def-85b6-7aa0b2c5e9ee-cert\") pod \"llmisvc-controller-manager-6466c554d8-bgs5p\" (UID: \"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:02.054496 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:02.054464 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flddd\" (UniqueName: \"kubernetes.io/projected/b793557f-8c8e-4def-85b6-7aa0b2c5e9ee-kube-api-access-flddd\") pod \"llmisvc-controller-manager-6466c554d8-bgs5p\" (UID: \"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:02.068411 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:06:02.068324 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:06:02.091082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:02.091058 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:02.461470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:02.461453 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p"] Oct 31 01:06:02.464054 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:06:02.464023 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb793557f_8c8e_4def_85b6_7aa0b2c5e9ee.slice/crio-6085af383a1ea3d031c7bd1e66166ba3363f28f1ae1e7af01e169a35f30f0372 WatchSource:0}: Error finding container 6085af383a1ea3d031c7bd1e66166ba3363f28f1ae1e7af01e169a35f30f0372: Status 404 returned error can't find the container with id 6085af383a1ea3d031c7bd1e66166ba3363f28f1ae1e7af01e169a35f30f0372 Oct 31 01:06:02.684308 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:02.684264 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" event={"ID":"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee","Type":"ContainerStarted","Data":"6085af383a1ea3d031c7bd1e66166ba3363f28f1ae1e7af01e169a35f30f0372"} Oct 31 01:06:03.690563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:03.690522 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" event={"ID":"b793557f-8c8e-4def-85b6-7aa0b2c5e9ee","Type":"ContainerStarted","Data":"b9da93c6df500cd6bb9cad3f60412f0b406f19de30450b3f9877027e503cab2f"} Oct 31 01:06:03.690999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:03.690656 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:03.805514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:03.805458 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" podStartSLOduration=2.101529543 podStartE2EDuration="2.805444015s" totalImagesPullingTime="703.914472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:06:03.803963083 +0000 UTC m=+535.610930607" watchObservedRunningTime="2026-10-31 01:06:03.805444015 +0000 UTC m=+535.612411554" Oct 31 01:06:32.065682 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:06:32.065646 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:06:34.697157 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:34.697129 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-bgs5p" Oct 31 01:06:34.860793 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:34.860764 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q"] Oct 31 01:06:34.861046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:34.861018 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" podUID="ab4229c9-2845-4477-9773-93bbb475ff13" containerName="manager" containerID="cri-o://6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155" gracePeriod=30 Oct 31 01:06:35.108689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.108666 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:06:35.197685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.197652 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk\" (UniqueName: \"kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk\") pod \"ab4229c9-2845-4477-9773-93bbb475ff13\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " Oct 31 01:06:35.197830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.197722 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert\" (UniqueName: \"kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert\") pod \"ab4229c9-2845-4477-9773-93bbb475ff13\" (UID: \"ab4229c9-2845-4477-9773-93bbb475ff13\") " Oct 31 01:06:35.199888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.199845 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert" pod "ab4229c9-2845-4477-9773-93bbb475ff13" (UID: "ab4229c9-2845-4477-9773-93bbb475ff13"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:06:35.200049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.200026 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk" pod "ab4229c9-2845-4477-9773-93bbb475ff13" (UID: "ab4229c9-2845-4477-9773-93bbb475ff13"). InnerVolumeSpecName "kube-api-access-bvrvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:06:35.299259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.299227 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvrvk\" (UniqueName: \"kubernetes.io/projected/ab4229c9-2845-4477-9773-93bbb475ff13-kube-api-access-bvrvk\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:35.299259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.299258 5756 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab4229c9-2845-4477-9773-93bbb475ff13-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:06:35.842615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.842574 5756 generic.go:356] "Generic (PLEG): container finished" podID="ab4229c9-2845-4477-9773-93bbb475ff13" containerID="6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155" exitCode=0 Oct 31 01:06:35.843107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.842621 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" Oct 31 01:06:35.843107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.842657 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" event={"ID":"ab4229c9-2845-4477-9773-93bbb475ff13","Type":"ContainerDied","Data":"6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155"} Oct 31 01:06:35.843107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.842726 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q" event={"ID":"ab4229c9-2845-4477-9773-93bbb475ff13","Type":"ContainerDied","Data":"a47468b7aac16ba201c07715855c2b340fbf206e0a571166f7600a7cf75b28c7"} Oct 31 01:06:35.843107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.842750 5756 scope.go:122] "RemoveContainer" containerID="6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155" Oct 31 01:06:35.851892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.851875 5756 scope.go:122] "RemoveContainer" containerID="6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155" Oct 31 01:06:35.852198 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:06:35.852179 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155\": container with ID starting with 6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155 not found: ID does not exist" containerID="6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155" Oct 31 01:06:35.852251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:35.852209 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155"} err="failed to get container status \"6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155\": rpc error: code = NotFound desc = could not find container \"6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155\": container with ID starting with 6770e866acacb971a067563708582984f6c21a3f3c550a4b0fef9d194acad155 not found: ID does not exist" Oct 31 01:06:36.005986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:36.005953 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q"] Oct 31 01:06:36.011373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:36.011345 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-t2f7q"] Oct 31 01:06:36.310677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:06:36.310647 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4229c9-2845-4477-9773-93bbb475ff13" path="/var/lib/kubelet/pods/ab4229c9-2845-4477-9773-93bbb475ff13/volumes" Oct 31 01:07:02.066400 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:07:02.066361 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:07:32.065268 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:07:32.065231 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:07:45.274495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:07:45.274458 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:07:52.501310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:07:52.501277 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:08:02.066498 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:08:02.066460 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:08:07.991192 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:07.991154 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:08:13.600935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.600905 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4"] Oct 31 01:08:13.601320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.601287 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ab4229c9-2845-4477-9773-93bbb475ff13" containerName="manager" Oct 31 01:08:13.601320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.601297 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4229c9-2845-4477-9773-93bbb475ff13" containerName="manager" Oct 31 01:08:13.601387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.601382 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4229c9-2845-4477-9773-93bbb475ff13" containerName="manager" Oct 31 01:08:13.603251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.603237 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.706520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.706487 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4"] Oct 31 01:08:13.725155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725115 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725154 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725179 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725285 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725333 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tm95\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725463 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725491 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725520 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.725565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.725557 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.751518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.751488 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 01:08:13.751518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.751503 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x7gsx\"" Oct 31 01:08:13.826239 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826205 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826239 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826240 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826263 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826283 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826320 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826343 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826375 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826426 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826452 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tm95\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826803 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.826949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826914 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.827011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.826974 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.827052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.827041 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.827156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.827139 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.829424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.829391 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.829540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.829524 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.882620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.882550 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.882620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.882584 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tm95\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95\") pod \"maas-default-gateway-openshift-default-fc8c56547-6xjn4\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:13.914499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:13.914466 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:14.333936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:14.333906 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4"] Oct 31 01:08:14.340877 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:14.340848 5756 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:08:15.293699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:15.293664 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" event={"ID":"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d","Type":"ContainerStarted","Data":"3d462775affdf510d172457d5889af8ccae060ef98f7386359cb3f96aa4c8fb8"} Oct 31 01:08:18.251270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:18.251237 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:08:18.251653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:18.251309 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:08:18.251653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:18.251348 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:08:19.322589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:19.322551 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" event={"ID":"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d","Type":"ContainerStarted","Data":"2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67"} Oct 31 01:08:19.487732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:19.487671 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podStartSLOduration=2.578450417 podStartE2EDuration="6.487655272s" totalImagesPullingTime="3.909204855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:19.486786938 +0000 UTC m=+671.293754480" watchObservedRunningTime="2026-10-31 01:08:19.487655272 +0000 UTC m=+671.294622816" Oct 31 01:08:19.915598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:19.915566 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:19.917115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:19.917090 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 01:08:19.917180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:19.917143 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 01:08:20.915687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:20.915648 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 01:08:20.916138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:20.915699 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 01:08:21.915139 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:21.915106 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 01:08:21.915317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:21.915161 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 01:08:22.915246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:22.915214 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 01:08:22.915656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:22.915277 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 01:08:23.915803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:23.915765 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 01:08:23.916226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:23.915827 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 01:08:24.915501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:24.915463 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 01:08:24.915682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:24.915515 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 01:08:25.067603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:25.067569 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:08:26.468220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:26.468190 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:26.468672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:26.468599 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:26.469419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:08:26.469403 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:08:32.065731 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:08:32.065697 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:09:02.065526 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:09:02.065486 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:09:32.066197 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:09:32.066150 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:10:02.070439 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:10:02.070404 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:10:14.793302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:14.793272 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:10:22.112695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.112664 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml"] Oct 31 01:10:22.116106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.116090 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.158517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.158486 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.158670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.158536 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g2n\" (UniqueName: \"kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n\") pod \"payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.170793 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.170747 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-60c85e15\"" Oct 31 01:10:22.220194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.220159 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-60c85e15-dockercfg-wh7gl\"" Oct 31 01:10:22.221384 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.221365 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml"] Oct 31 01:10:22.259295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.259265 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.259385 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.259356 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g2n\" (UniqueName: \"kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n\") pod \"payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.259949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.259932 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume\") pod \"payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.316413 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.316378 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g2n\" (UniqueName: \"kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n\") pod \"payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.428528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.428458 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:22.798772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.798748 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml"] Oct 31 01:10:22.801668 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:10:22.801642 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0b80fc_1520_4a87_9c4f_26659defb898.slice/crio-678b65339ab25dda657535e6169cfd454ee2a5a7d0934bd92f45d3e63dc972aa WatchSource:0}: Error finding container 678b65339ab25dda657535e6169cfd454ee2a5a7d0934bd92f45d3e63dc972aa: Status 404 returned error can't find the container with id 678b65339ab25dda657535e6169cfd454ee2a5a7d0934bd92f45d3e63dc972aa Oct 31 01:10:22.920707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:22.920668 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" event={"ID":"7f0b80fc-1520-4a87-9c4f-26659defb898","Type":"ContainerStarted","Data":"678b65339ab25dda657535e6169cfd454ee2a5a7d0934bd92f45d3e63dc972aa"} Oct 31 01:10:23.196842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:23.196750 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:10:25.938088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:25.938054 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" event={"ID":"7f0b80fc-1520-4a87-9c4f-26659defb898","Type":"ContainerStarted","Data":"d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a"} Oct 31 01:10:25.938472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:25.938190 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:26.054351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:26.054291 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" podStartSLOduration=1.353278318 podStartE2EDuration="4.0542724s" totalImagesPullingTime="2.700994082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:26.053532188 +0000 UTC m=+797.860499731" watchObservedRunningTime="2026-10-31 01:10:26.0542724 +0000 UTC m=+797.861239944" Oct 31 01:10:30.078778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.078744 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz"] Oct 31 01:10:30.082439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.082421 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.134297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.134266 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:10:30.135680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.135654 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.135794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.135688 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.135794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.135721 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7rl\" (UniqueName: \"kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.183552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.183513 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:10:30.185107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.185079 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz"] Oct 31 01:10:30.237254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.237223 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.237459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.237262 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.237459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.237296 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7rl\" (UniqueName: \"kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.238019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.237996 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.239693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.239674 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.294017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.293987 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7rl\" (UniqueName: \"kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl\") pod \"maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.395047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.394960 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:30.765242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.765167 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz"] Oct 31 01:10:30.768174 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:10:30.768146 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eba785_f4f2_4142_84c9_609837a84bf2.slice/crio-12accb9bd804b9622d2c9ed8a9828d772f604128f5d384929666aaf2e0842448 WatchSource:0}: Error finding container 12accb9bd804b9622d2c9ed8a9828d772f604128f5d384929666aaf2e0842448: Status 404 returned error can't find the container with id 12accb9bd804b9622d2c9ed8a9828d772f604128f5d384929666aaf2e0842448 Oct 31 01:10:30.963254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:30.963218 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" event={"ID":"c9eba785-f4f2-4142-84c9-609837a84bf2","Type":"ContainerStarted","Data":"12accb9bd804b9622d2c9ed8a9828d772f604128f5d384929666aaf2e0842448"} Oct 31 01:10:32.069774 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:10:32.069735 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:10:32.115322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:32.115291 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:10:35.988541 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:35.988465 5756 generic.go:356] "Generic (PLEG): container finished" podID="c9eba785-f4f2-4142-84c9-609837a84bf2" containerID="f373b4d2ceaf7ff7335f37b9be84c08b63eb5f9d5ecb7ed5744b9fe3e4ae12fd" exitCode=0 Oct 31 01:10:35.988944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:35.988540 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" event={"ID":"c9eba785-f4f2-4142-84c9-609837a84bf2","Type":"ContainerDied","Data":"f373b4d2ceaf7ff7335f37b9be84c08b63eb5f9d5ecb7ed5744b9fe3e4ae12fd"} Oct 31 01:10:36.945728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:36.945698 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:10:37.130332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.130309 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:37.207073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.206989 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl\" (UniqueName: \"kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl\") pod \"c9eba785-f4f2-4142-84c9-609837a84bf2\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " Oct 31 01:10:37.207073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.207053 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca\") pod \"c9eba785-f4f2-4142-84c9-609837a84bf2\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " Oct 31 01:10:37.207292 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.207133 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca\") pod \"c9eba785-f4f2-4142-84c9-609837a84bf2\" (UID: \"c9eba785-f4f2-4142-84c9-609837a84bf2\") " Oct 31 01:10:37.207456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.207433 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca" pod "c9eba785-f4f2-4142-84c9-609837a84bf2" (UID: "c9eba785-f4f2-4142-84c9-609837a84bf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:37.209423 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.209402 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca" pod "c9eba785-f4f2-4142-84c9-609837a84bf2" (UID: "c9eba785-f4f2-4142-84c9-609837a84bf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:37.209541 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.209425 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl" pod "c9eba785-f4f2-4142-84c9-609837a84bf2" (UID: "c9eba785-f4f2-4142-84c9-609837a84bf2"). InnerVolumeSpecName "kube-api-access-mk7rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:37.308023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.307988 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk7rl\" (UniqueName: \"kubernetes.io/projected/c9eba785-f4f2-4142-84c9-609837a84bf2-kube-api-access-mk7rl\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.308023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.308018 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.308023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.308029 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eba785-f4f2-4142-84c9-609837a84bf2-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:37.814604 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.814574 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:10:37.999233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.999207 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" Oct 31 01:10:37.999398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.999246 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz" event={"ID":"c9eba785-f4f2-4142-84c9-609837a84bf2","Type":"ContainerDied","Data":"12accb9bd804b9622d2c9ed8a9828d772f604128f5d384929666aaf2e0842448"} Oct 31 01:10:37.999398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:37.999293 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12accb9bd804b9622d2c9ed8a9828d772f604128f5d384929666aaf2e0842448" Oct 31 01:10:46.285215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:46.285140 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:10:52.522153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:52.522115 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz"] Oct 31 01:10:52.571418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:52.571384 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-20f0d1b6-5swjz"] Oct 31 01:10:54.312491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:54.312453 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9eba785-f4f2-4142-84c9-609837a84bf2" path="/var/lib/kubelet/pods/c9eba785-f4f2-4142-84c9-609837a84bf2/volumes" Oct 31 01:10:55.659500 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.659466 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8"] Oct 31 01:10:55.660104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.660082 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c9eba785-f4f2-4142-84c9-609837a84bf2" containerName="revoke-keys" Oct 31 01:10:55.660193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.660105 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9eba785-f4f2-4142-84c9-609837a84bf2" containerName="revoke-keys" Oct 31 01:10:55.660244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.660222 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9eba785-f4f2-4142-84c9-609837a84bf2" containerName="revoke-keys" Oct 31 01:10:55.663400 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.663380 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.682735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.682710 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td58h\" (UniqueName: \"kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.682886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.682774 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.682955 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.682913 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.766705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.766671 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:10:55.766856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.766776 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:10:55.768119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.768103 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8"] Oct 31 01:10:55.784155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.784112 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.784325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.784209 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.784325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.784273 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td58h\" (UniqueName: \"kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.784906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.784887 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.786517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.786501 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.840908 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.840852 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td58h\" (UniqueName: \"kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h\") pod \"maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:55.976036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:55.975960 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:56.347370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:56.347345 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8"] Oct 31 01:10:56.350158 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:10:56.350131 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043b56cb_e210_4b6c_9209_65c3f0b64234.slice/crio-a54c496fcc4b6f2d0f6dbf0abf718b426d1ffcf415cf322f53322e26676a3db2 WatchSource:0}: Error finding container a54c496fcc4b6f2d0f6dbf0abf718b426d1ffcf415cf322f53322e26676a3db2: Status 404 returned error can't find the container with id a54c496fcc4b6f2d0f6dbf0abf718b426d1ffcf415cf322f53322e26676a3db2 Oct 31 01:10:57.088592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:57.088555 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" event={"ID":"043b56cb-e210-4b6c-9209-65c3f0b64234","Type":"ContainerStarted","Data":"f52be11ded565899d82eb7c8e42fb6b33bab2924ff7df11ac8ccfcaa05b1a03a"} Oct 31 01:10:57.089026 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:57.088607 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" event={"ID":"043b56cb-e210-4b6c-9209-65c3f0b64234","Type":"ContainerStarted","Data":"a54c496fcc4b6f2d0f6dbf0abf718b426d1ffcf415cf322f53322e26676a3db2"} Oct 31 01:10:57.201834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:57.201787 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" podStartSLOduration=2.201773855 podStartE2EDuration="2.201773855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:57.200824907 +0000 UTC m=+829.007792462" watchObservedRunningTime="2026-10-31 01:10:57.201773855 +0000 UTC m=+829.008741397" Oct 31 01:10:58.095106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:58.095069 5756 generic.go:356] "Generic (PLEG): container finished" podID="043b56cb-e210-4b6c-9209-65c3f0b64234" containerID="f52be11ded565899d82eb7c8e42fb6b33bab2924ff7df11ac8ccfcaa05b1a03a" exitCode=0 Oct 31 01:10:58.095510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:58.095118 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" event={"ID":"043b56cb-e210-4b6c-9209-65c3f0b64234","Type":"ContainerDied","Data":"f52be11ded565899d82eb7c8e42fb6b33bab2924ff7df11ac8ccfcaa05b1a03a"} Oct 31 01:10:59.240066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.240034 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:10:59.320530 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.320501 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca\") pod \"043b56cb-e210-4b6c-9209-65c3f0b64234\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " Oct 31 01:10:59.320685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.320562 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h\" (UniqueName: \"kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h\") pod \"043b56cb-e210-4b6c-9209-65c3f0b64234\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " Oct 31 01:10:59.320685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.320610 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca\") pod \"043b56cb-e210-4b6c-9209-65c3f0b64234\" (UID: \"043b56cb-e210-4b6c-9209-65c3f0b64234\") " Oct 31 01:10:59.320928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.320906 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca" pod "043b56cb-e210-4b6c-9209-65c3f0b64234" (UID: "043b56cb-e210-4b6c-9209-65c3f0b64234"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:10:59.322690 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.322663 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h" pod "043b56cb-e210-4b6c-9209-65c3f0b64234" (UID: "043b56cb-e210-4b6c-9209-65c3f0b64234"). InnerVolumeSpecName "kube-api-access-td58h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:10:59.322810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.322769 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca" pod "043b56cb-e210-4b6c-9209-65c3f0b64234" (UID: "043b56cb-e210-4b6c-9209-65c3f0b64234"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:10:59.422435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.422351 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:59.422435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.422379 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td58h\" (UniqueName: \"kubernetes.io/projected/043b56cb-e210-4b6c-9209-65c3f0b64234-kube-api-access-td58h\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:59.422435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.422390 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/043b56cb-e210-4b6c-9209-65c3f0b64234-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:10:59.814326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.814293 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69"] Oct 31 01:10:59.814721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.814709 5756 cpu_manager.go:414] "containerMap: removing container" podUID="043b56cb-e210-4b6c-9209-65c3f0b64234" containerName="revoke-keys" Oct 31 01:10:59.814772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.814722 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043b56cb-e210-4b6c-9209-65c3f0b64234" containerName="revoke-keys" Oct 31 01:10:59.814806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.814779 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="043b56cb-e210-4b6c-9209-65c3f0b64234" containerName="revoke-keys" Oct 31 01:10:59.823254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.823221 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:10:59.874667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.874635 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69"] Oct 31 01:10:59.926616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.926580 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfpq4\" (UniqueName: \"kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:10:59.926816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.926751 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:10:59.926816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:10:59.926804 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.028209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.028148 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfpq4\" (UniqueName: \"kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.028362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.028308 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.028362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.028349 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.029043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.028977 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.031204 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.031178 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.084519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.084455 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfpq4\" (UniqueName: \"kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4\") pod \"maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.107049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.107024 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" Oct 31 01:11:00.107197 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.107025 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8" event={"ID":"043b56cb-e210-4b6c-9209-65c3f0b64234","Type":"ContainerDied","Data":"a54c496fcc4b6f2d0f6dbf0abf718b426d1ffcf415cf322f53322e26676a3db2"} Oct 31 01:11:00.107197 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.107138 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a54c496fcc4b6f2d0f6dbf0abf718b426d1ffcf415cf322f53322e26676a3db2" Oct 31 01:11:00.136055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.136028 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:00.508276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:00.508249 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69"] Oct 31 01:11:00.511164 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:11:00.511134 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e4e2ec_eea7_4f33_b15a_f465a921fd63.slice/crio-84bc2be730e8b92623c7447f24d871a988f068cbe5e07dcfae08612b26ba356f WatchSource:0}: Error finding container 84bc2be730e8b92623c7447f24d871a988f068cbe5e07dcfae08612b26ba356f: Status 404 returned error can't find the container with id 84bc2be730e8b92623c7447f24d871a988f068cbe5e07dcfae08612b26ba356f Oct 31 01:11:01.115531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:01.115435 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" event={"ID":"96e4e2ec-eea7-4f33-b15a-f465a921fd63","Type":"ContainerStarted","Data":"1fa63275a0ebf252f61cd498b3402f860f283e0a559bb110ec0918dabf2381f5"} Oct 31 01:11:01.115531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:01.115500 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" event={"ID":"96e4e2ec-eea7-4f33-b15a-f465a921fd63","Type":"ContainerStarted","Data":"84bc2be730e8b92623c7447f24d871a988f068cbe5e07dcfae08612b26ba356f"} Oct 31 01:11:01.229228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:01.229165 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" podStartSLOduration=2.22914701 podStartE2EDuration="2.22914701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:01.228382328 +0000 UTC m=+833.035349933" watchObservedRunningTime="2026-10-31 01:11:01.22914701 +0000 UTC m=+833.036114555" Oct 31 01:11:02.066336 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:11:02.066301 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:11:02.121561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:02.121450 5756 generic.go:356] "Generic (PLEG): container finished" podID="96e4e2ec-eea7-4f33-b15a-f465a921fd63" containerID="1fa63275a0ebf252f61cd498b3402f860f283e0a559bb110ec0918dabf2381f5" exitCode=0 Oct 31 01:11:02.121561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:02.121534 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" event={"ID":"96e4e2ec-eea7-4f33-b15a-f465a921fd63","Type":"ContainerDied","Data":"1fa63275a0ebf252f61cd498b3402f860f283e0a559bb110ec0918dabf2381f5"} Oct 31 01:11:03.264386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.264362 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:03.361845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.361810 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca\") pod \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " Oct 31 01:11:03.362065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.361945 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca\") pod \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " Oct 31 01:11:03.362065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.361982 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4\" (UniqueName: \"kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4\") pod \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\" (UID: \"96e4e2ec-eea7-4f33-b15a-f465a921fd63\") " Oct 31 01:11:03.362248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.362194 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca" pod "96e4e2ec-eea7-4f33-b15a-f465a921fd63" (UID: "96e4e2ec-eea7-4f33-b15a-f465a921fd63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:03.362307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.362282 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:03.364206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.364176 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca" pod "96e4e2ec-eea7-4f33-b15a-f465a921fd63" (UID: "96e4e2ec-eea7-4f33-b15a-f465a921fd63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:03.364319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.364300 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4" pod "96e4e2ec-eea7-4f33-b15a-f465a921fd63" (UID: "96e4e2ec-eea7-4f33-b15a-f465a921fd63"). InnerVolumeSpecName "kube-api-access-sfpq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:03.463629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.463549 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e4e2ec-eea7-4f33-b15a-f465a921fd63-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:03.463629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.463577 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfpq4\" (UniqueName: \"kubernetes.io/projected/96e4e2ec-eea7-4f33-b15a-f465a921fd63-kube-api-access-sfpq4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:03.686579 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:03.686540 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:11:04.132695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.132662 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" Oct 31 01:11:04.132907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.132665 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69" event={"ID":"96e4e2ec-eea7-4f33-b15a-f465a921fd63","Type":"ContainerDied","Data":"84bc2be730e8b92623c7447f24d871a988f068cbe5e07dcfae08612b26ba356f"} Oct 31 01:11:04.132907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.132788 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84bc2be730e8b92623c7447f24d871a988f068cbe5e07dcfae08612b26ba356f" Oct 31 01:11:04.787358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.787325 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l"] Oct 31 01:11:04.787906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.787888 5756 cpu_manager.go:414] "containerMap: removing container" podUID="96e4e2ec-eea7-4f33-b15a-f465a921fd63" containerName="revoke-keys" Oct 31 01:11:04.787906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.787906 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e4e2ec-eea7-4f33-b15a-f465a921fd63" containerName="revoke-keys" Oct 31 01:11:04.788025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.787988 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e4e2ec-eea7-4f33-b15a-f465a921fd63" containerName="revoke-keys" Oct 31 01:11:04.806525 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.806499 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.847478 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.847435 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l"] Oct 31 01:11:04.875722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.875692 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.875879 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.875799 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8qw\" (UniqueName: \"kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.875879 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.875831 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.894936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.894916 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:11:04.895077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.894949 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:11:04.977141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.977107 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.977332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.977178 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8qw\" (UniqueName: \"kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.977332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.977197 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.977740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.977716 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:04.979633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:04.979613 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:05.033519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:05.033486 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8qw\" (UniqueName: \"kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw\") pod \"maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:05.129911 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:05.129813 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:05.506910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:05.506881 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l"] Oct 31 01:11:05.509752 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:11:05.509723 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87afa201_63de_405c_9c6b_7b7b70e19fa0.slice/crio-a4ba6d4a16f5cd294af540d27bace5ad5512d75f455cde94a0cd24817425e81d WatchSource:0}: Error finding container a4ba6d4a16f5cd294af540d27bace5ad5512d75f455cde94a0cd24817425e81d: Status 404 returned error can't find the container with id a4ba6d4a16f5cd294af540d27bace5ad5512d75f455cde94a0cd24817425e81d Oct 31 01:11:06.145729 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:06.145633 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" event={"ID":"87afa201-63de-405c-9c6b-7b7b70e19fa0","Type":"ContainerStarted","Data":"2bd781ccb2531bd1b82e8003503bb917e059540366b931f6a49f490e9e6338cf"} Oct 31 01:11:06.145729 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:06.145684 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" event={"ID":"87afa201-63de-405c-9c6b-7b7b70e19fa0","Type":"ContainerStarted","Data":"a4ba6d4a16f5cd294af540d27bace5ad5512d75f455cde94a0cd24817425e81d"} Oct 31 01:11:06.258277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:06.258224 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" podStartSLOduration=2.258207938 podStartE2EDuration="2.258207938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:06.257344631 +0000 UTC m=+838.064312175" watchObservedRunningTime="2026-10-31 01:11:06.258207938 +0000 UTC m=+838.065175481" Oct 31 01:11:07.151410 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:07.151379 5756 generic.go:356] "Generic (PLEG): container finished" podID="87afa201-63de-405c-9c6b-7b7b70e19fa0" containerID="2bd781ccb2531bd1b82e8003503bb917e059540366b931f6a49f490e9e6338cf" exitCode=0 Oct 31 01:11:07.151797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:07.151467 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" event={"ID":"87afa201-63de-405c-9c6b-7b7b70e19fa0","Type":"ContainerDied","Data":"2bd781ccb2531bd1b82e8003503bb917e059540366b931f6a49f490e9e6338cf"} Oct 31 01:11:08.290472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.290449 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:08.413362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.411966 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca\") pod \"87afa201-63de-405c-9c6b-7b7b70e19fa0\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " Oct 31 01:11:08.413362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.412112 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw\" (UniqueName: \"kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw\") pod \"87afa201-63de-405c-9c6b-7b7b70e19fa0\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " Oct 31 01:11:08.413362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.412177 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca\") pod \"87afa201-63de-405c-9c6b-7b7b70e19fa0\" (UID: \"87afa201-63de-405c-9c6b-7b7b70e19fa0\") " Oct 31 01:11:08.413362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.412311 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca" pod "87afa201-63de-405c-9c6b-7b7b70e19fa0" (UID: "87afa201-63de-405c-9c6b-7b7b70e19fa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:08.413362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.412569 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:08.415258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.415234 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca" pod "87afa201-63de-405c-9c6b-7b7b70e19fa0" (UID: "87afa201-63de-405c-9c6b-7b7b70e19fa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:08.415451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.415431 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw" pod "87afa201-63de-405c-9c6b-7b7b70e19fa0" (UID: "87afa201-63de-405c-9c6b-7b7b70e19fa0"). InnerVolumeSpecName "kube-api-access-tl8qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:08.513776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.513743 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8qw\" (UniqueName: \"kubernetes.io/projected/87afa201-63de-405c-9c6b-7b7b70e19fa0-kube-api-access-tl8qw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:08.513776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:08.513771 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87afa201-63de-405c-9c6b-7b7b70e19fa0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:09.162176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:09.162146 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" Oct 31 01:11:09.162176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:09.162160 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l" event={"ID":"87afa201-63de-405c-9c6b-7b7b70e19fa0","Type":"ContainerDied","Data":"a4ba6d4a16f5cd294af540d27bace5ad5512d75f455cde94a0cd24817425e81d"} Oct 31 01:11:09.162384 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:09.162229 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4ba6d4a16f5cd294af540d27bace5ad5512d75f455cde94a0cd24817425e81d" Oct 31 01:11:17.683979 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:17.683945 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:11:20.863012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:20.862977 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69"] Oct 31 01:11:20.918073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:20.914214 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3d4f9b87-99z69"] Oct 31 01:11:21.251746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.251663 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9"] Oct 31 01:11:21.252168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.252151 5756 cpu_manager.go:414] "containerMap: removing container" podUID="87afa201-63de-405c-9c6b-7b7b70e19fa0" containerName="revoke-keys" Oct 31 01:11:21.252221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.252171 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87afa201-63de-405c-9c6b-7b7b70e19fa0" containerName="revoke-keys" Oct 31 01:11:21.252271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.252260 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="87afa201-63de-405c-9c6b-7b7b70e19fa0" containerName="revoke-keys" Oct 31 01:11:21.254347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.254334 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.306837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.306809 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:11:21.335305 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.335265 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.335486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.335319 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.335486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.335401 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffft\" (UniqueName: \"kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.356048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.356017 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:11:21.357493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.357474 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9"] Oct 31 01:11:21.436128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.436100 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.436287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.436146 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffft\" (UniqueName: \"kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.436287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.436222 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.436918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.436838 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.438656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.438633 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.492650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.492625 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffft\" (UniqueName: \"kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft\") pod \"maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.566641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.566609 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:21.977524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:21.977499 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9"] Oct 31 01:11:21.980424 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:11:21.980398 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cc0ed3_a79c_45ae_8ded_f094f46dbfdd.slice/crio-d8762f428f8959028b121b28803275aaceb22e8ed93334a2ea831f6920351393 WatchSource:0}: Error finding container d8762f428f8959028b121b28803275aaceb22e8ed93334a2ea831f6920351393: Status 404 returned error can't find the container with id d8762f428f8959028b121b28803275aaceb22e8ed93334a2ea831f6920351393 Oct 31 01:11:22.228080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:22.228000 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" event={"ID":"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd","Type":"ContainerStarted","Data":"d8762f428f8959028b121b28803275aaceb22e8ed93334a2ea831f6920351393"} Oct 31 01:11:22.309796 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:22.309762 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e4e2ec-eea7-4f33-b15a-f465a921fd63" path="/var/lib/kubelet/pods/96e4e2ec-eea7-4f33-b15a-f465a921fd63/volumes" Oct 31 01:11:23.234002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:23.233925 5756 generic.go:356] "Generic (PLEG): container finished" podID="37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" containerID="cbe9e9f7b4a1912d233d45ed2966717ba42ae65ae538cdf0893e2818f6153afa" exitCode=0 Oct 31 01:11:23.234377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:23.234018 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" event={"ID":"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd","Type":"ContainerDied","Data":"cbe9e9f7b4a1912d233d45ed2966717ba42ae65ae538cdf0893e2818f6153afa"} Oct 31 01:11:24.374725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.374703 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:24.478434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.478405 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft\" (UniqueName: \"kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft\") pod \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " Oct 31 01:11:24.478612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.478590 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca\") pod \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " Oct 31 01:11:24.478654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.478631 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca\") pod \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\" (UID: \"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd\") " Oct 31 01:11:24.479153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.479096 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca" pod "37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" (UID: "37cc0ed3-a79c-45ae-8ded-f094f46dbfdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:24.481054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.481027 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft" pod "37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" (UID: "37cc0ed3-a79c-45ae-8ded-f094f46dbfdd"). InnerVolumeSpecName "kube-api-access-8ffft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:24.481163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.481080 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca" pod "37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" (UID: "37cc0ed3-a79c-45ae-8ded-f094f46dbfdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:11:24.580062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.580026 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:24.580062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.580055 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:24.580062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:24.580066 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffft\" (UniqueName: \"kubernetes.io/projected/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd-kube-api-access-8ffft\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:25.245097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:25.245063 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" event={"ID":"37cc0ed3-a79c-45ae-8ded-f094f46dbfdd","Type":"ContainerDied","Data":"d8762f428f8959028b121b28803275aaceb22e8ed93334a2ea831f6920351393"} Oct 31 01:11:25.245263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:25.245116 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8762f428f8959028b121b28803275aaceb22e8ed93334a2ea831f6920351393" Oct 31 01:11:25.245263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:25.245074 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9" Oct 31 01:11:29.074365 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:29.074331 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:11:32.067974 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:11:32.067838 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:11:32.532493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:32.532464 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml"] Oct 31 01:11:36.139129 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:36.139098 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8"] Oct 31 01:11:36.188255 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:36.188225 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-79f7219a-8mwt8"] Oct 31 01:11:36.310135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:36.310101 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043b56cb-e210-4b6c-9209-65c3f0b64234" path="/var/lib/kubelet/pods/043b56cb-e210-4b6c-9209-65c3f0b64234/volumes" Oct 31 01:11:37.533280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.533243 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l"] Oct 31 01:11:37.533745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.533280 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" podUID="7f0b80fc-1520-4a87-9c4f-26659defb898" containerName="payload-processing" containerID="cri-o://d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a" gracePeriod=25 Oct 31 01:11:37.583053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.583025 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-935bafe7-5sz5l"] Oct 31 01:11:37.775186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.775165 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:11:37.795647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.795585 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume\") pod \"7f0b80fc-1520-4a87-9c4f-26659defb898\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " Oct 31 01:11:37.795781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.795720 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n\" (UniqueName: \"kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n\") pod \"7f0b80fc-1520-4a87-9c4f-26659defb898\" (UID: \"7f0b80fc-1520-4a87-9c4f-26659defb898\") " Oct 31 01:11:37.796046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.796022 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume" pod "7f0b80fc-1520-4a87-9c4f-26659defb898" (UID: "7f0b80fc-1520-4a87-9c4f-26659defb898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:11:37.798089 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.798057 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n" pod "7f0b80fc-1520-4a87-9c4f-26659defb898" (UID: "7f0b80fc-1520-4a87-9c4f-26659defb898"). InnerVolumeSpecName "kube-api-access-p8g2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:11:37.896902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.896834 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0b80fc-1520-4a87-9c4f-26659defb898-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:37.896902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:37.896878 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8g2n\" (UniqueName: \"kubernetes.io/projected/7f0b80fc-1520-4a87-9c4f-26659defb898-kube-api-access-p8g2n\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:11:38.309993 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.309963 5756 generic.go:356] "Generic (PLEG): container finished" podID="7f0b80fc-1520-4a87-9c4f-26659defb898" containerID="d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a" exitCode=0 Oct 31 01:11:38.310163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.310038 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" Oct 31 01:11:38.310207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.310169 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87afa201-63de-405c-9c6b-7b7b70e19fa0" path="/var/lib/kubelet/pods/87afa201-63de-405c-9c6b-7b7b70e19fa0/volumes" Oct 31 01:11:38.310583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.310563 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" event={"ID":"7f0b80fc-1520-4a87-9c4f-26659defb898","Type":"ContainerDied","Data":"d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a"} Oct 31 01:11:38.310662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.310600 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml" event={"ID":"7f0b80fc-1520-4a87-9c4f-26659defb898","Type":"ContainerDied","Data":"678b65339ab25dda657535e6169cfd454ee2a5a7d0934bd92f45d3e63dc972aa"} Oct 31 01:11:38.310662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.310622 5756 scope.go:122] "RemoveContainer" containerID="d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a" Oct 31 01:11:38.325581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.325564 5756 scope.go:122] "RemoveContainer" containerID="d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a" Oct 31 01:11:38.325916 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:11:38.325895 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a\": container with ID starting with d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a not found: ID does not exist" containerID="d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a" Oct 31 01:11:38.325975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.325928 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a"} err="failed to get container status \"d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a\": rpc error: code = NotFound desc = could not find container \"d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a\": container with ID starting with d1a202aa9e1a35ac5b6b0fa6402c65e755a49f9aa62c555a5d85eaac1422d96a not found: ID does not exist" Oct 31 01:11:38.471959 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.471914 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml"] Oct 31 01:11:38.520758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:38.520724 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-60c85e15-749d76b855-zk9ml"] Oct 31 01:11:39.000225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000188 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc"] Oct 31 01:11:39.000716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000687 5756 cpu_manager.go:414] "containerMap: removing container" podUID="7f0b80fc-1520-4a87-9c4f-26659defb898" containerName="payload-processing" Oct 31 01:11:39.000716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000717 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0b80fc-1520-4a87-9c4f-26659defb898" containerName="payload-processing" Oct 31 01:11:39.000854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000733 5756 cpu_manager.go:414] "containerMap: removing container" podUID="37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" containerName="revoke-keys" Oct 31 01:11:39.000854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000738 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" containerName="revoke-keys" Oct 31 01:11:39.000988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000856 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0b80fc-1520-4a87-9c4f-26659defb898" containerName="payload-processing" Oct 31 01:11:39.000988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.000888 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" containerName="revoke-keys" Oct 31 01:11:39.003647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.003628 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.055789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.055758 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e621db-openshift-default-dockercfg-p8kgb\"" Oct 31 01:11:39.106698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.106663 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc"] Oct 31 01:11:39.107449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107425 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107465 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vngdc\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107489 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107551 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107683 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107738 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107846 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.107971 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.107947 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.108043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.108016 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209185 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209234 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209268 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209287 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209308 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vngdc\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209331 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209352 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209371 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209410 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209707 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209908 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.209990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.209937 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.210177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.210144 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.210777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.210735 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.212465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.212438 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.212663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.212645 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.266526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.266448 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.266660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.266551 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vngdc\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.314620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.314584 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:39.737994 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.737919 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc"] Oct 31 01:11:39.740826 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:11:39.740796 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb1af39_ea05_4b0b_8a83_90bb2257fcf0.slice/crio-e64cd841541200059cce86ecc86ce09c8903e11bf272554753606511ec02095a WatchSource:0}: Error finding container e64cd841541200059cce86ecc86ce09c8903e11bf272554753606511ec02095a: Status 404 returned error can't find the container with id e64cd841541200059cce86ecc86ce09c8903e11bf272554753606511ec02095a Oct 31 01:11:39.742952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.742917 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:11:39.743075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.742996 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:11:39.743075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:39.743036 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:11:40.310548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:40.310512 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0b80fc-1520-4a87-9c4f-26659defb898" path="/var/lib/kubelet/pods/7f0b80fc-1520-4a87-9c4f-26659defb898/volumes" Oct 31 01:11:40.321712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:40.321685 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" event={"ID":"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0","Type":"ContainerStarted","Data":"d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872"} Oct 31 01:11:40.321855 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:40.321733 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" event={"ID":"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0","Type":"ContainerStarted","Data":"e64cd841541200059cce86ecc86ce09c8903e11bf272554753606511ec02095a"} Oct 31 01:11:40.439994 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:40.439687 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podStartSLOduration=2.43967011 podStartE2EDuration="2.43967011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:40.439452346 +0000 UTC m=+872.246419902" watchObservedRunningTime="2026-10-31 01:11:40.43967011 +0000 UTC m=+872.246637653" Oct 31 01:11:41.315397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:41.315366 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:41.321004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:41.320977 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:41.326198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:41.326175 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:41.326967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:41.326950 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:11:43.836913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:43.836882 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9"] Oct 31 01:11:43.886104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:43.886078 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-60c85e15-zc6p9"] Oct 31 01:11:44.310013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:44.309980 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cc0ed3-a79c-45ae-8ded-f094f46dbfdd" path="/var/lib/kubelet/pods/37cc0ed3-a79c-45ae-8ded-f094f46dbfdd/volumes" Oct 31 01:11:48.273764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:48.273728 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:11:54.138082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.138039 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz"] Oct 31 01:11:54.140735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.140718 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.192471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.192436 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-acd385-openshift-default-dockercfg-76xk7\"" Oct 31 01:11:54.243574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.243538 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz"] Oct 31 01:11:54.255216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255190 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255226 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255244 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255325 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255373 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255459 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255613 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255770 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.255918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.255823 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357331 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357380 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357402 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357524 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357582 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357608 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357655 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357776 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.357828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357817 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.358029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.357922 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.358088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.358029 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.358162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.358139 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.358227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.358210 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.358740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.358717 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.359921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.359895 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.360028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.360012 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.413692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.413625 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.413853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.413836 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.453782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.453751 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:54.910301 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.910276 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz"] Oct 31 01:11:54.912790 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:11:54.912760 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70c7134_e696_4718_b986_94ccb0d1f0bc.slice/crio-e46ef00a1942e09ed74ae35b1667ebacc04251d6366ae50f713f5ec76fb60ef8 WatchSource:0}: Error finding container e46ef00a1942e09ed74ae35b1667ebacc04251d6366ae50f713f5ec76fb60ef8: Status 404 returned error can't find the container with id e46ef00a1942e09ed74ae35b1667ebacc04251d6366ae50f713f5ec76fb60ef8 Oct 31 01:11:54.918355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.918317 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:11:54.918477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.918386 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:11:54.918477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:54.918434 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:11:55.394412 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:55.394105 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" event={"ID":"c70c7134-e696-4718-b986-94ccb0d1f0bc","Type":"ContainerStarted","Data":"086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6"} Oct 31 01:11:55.394412 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:55.394172 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" event={"ID":"c70c7134-e696-4718-b986-94ccb0d1f0bc","Type":"ContainerStarted","Data":"e46ef00a1942e09ed74ae35b1667ebacc04251d6366ae50f713f5ec76fb60ef8"} Oct 31 01:11:55.457259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:55.457233 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:11:55.460126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:55.460093 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:11:55.460290 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:55.460143 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 01:11:55.518570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:55.518518 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podStartSLOduration=1.518504396 podStartE2EDuration="1.518504396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:55.517421587 +0000 UTC m=+887.324389131" watchObservedRunningTime="2026-10-31 01:11:55.518504396 +0000 UTC m=+887.325471941" Oct 31 01:11:56.455093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:56.455058 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:11:56.455646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:56.455108 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 01:11:57.454769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:57.454736 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:11:57.454952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:57.454788 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 01:11:58.454335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:58.454300 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:11:58.454749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:58.454352 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 01:11:59.454254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:59.454217 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:11:59.454444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:11:59.454271 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 01:12:00.455290 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:00.455251 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 31 01:12:00.455698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:00.455321 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 31 01:12:01.460312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:01.459876 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:12:01.460808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:01.460338 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:12:01.461254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:01.461224 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:12:02.067071 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:12:02.067026 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:12:08.674624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:08.674543 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:12:10.265015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.264981 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg"] Oct 31 01:12:10.268269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.268250 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.322855 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.322825 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-26fbc4-openshift-default-dockercfg-bscl8\"" Oct 31 01:12:10.366406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.366373 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg"] Oct 31 01:12:10.421814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.421778 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.421992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.421820 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422011 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422040 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422066 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvtz\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422097 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422165 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422230 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.422286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.422265 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523203 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvtz\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523244 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523264 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523281 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523296 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523317 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523342 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523479 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523507 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523660 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.523847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.523735 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.524200 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.524176 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.524297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.524252 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.524443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.524423 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.525996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.525979 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.526108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.526091 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.580499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.580473 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.580643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.580536 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvtz\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:10.880351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:10.880322 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:11.337575 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.337545 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg"] Oct 31 01:12:11.350095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.349026 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:12:11.350095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.349118 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:12:11.350095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.349162 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:12:11.481472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.481437 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" event={"ID":"ec648a48-3742-4497-8e93-a52e12d689b0","Type":"ContainerStarted","Data":"6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1"} Oct 31 01:12:11.481674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.481499 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" event={"ID":"ec648a48-3742-4497-8e93-a52e12d689b0","Type":"ContainerStarted","Data":"e8c7400f815c7f2c47e90607f6eecc099623a535914110a9289edff15e2ccbee"} Oct 31 01:12:11.600195 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.600090 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podStartSLOduration=1.60007541 podStartE2EDuration="1.60007541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:11.598721515 +0000 UTC m=+903.405689098" watchObservedRunningTime="2026-10-31 01:12:11.60007541 +0000 UTC m=+903.407042953" Oct 31 01:12:11.884986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:11.880996 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:12.882459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:12.882413 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:12:12.882898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:12.882478 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 01:12:13.881166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:13.881127 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:12:13.881459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:13.881430 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 01:12:14.881181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:14.881146 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:12:14.881772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:14.881198 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 01:12:15.881718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:15.881569 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:12:15.881718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:15.881635 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 01:12:16.881092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:16.881050 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 31 01:12:16.881300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:16.881114 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 31 01:12:17.476518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.476474 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t"] Oct 31 01:12:17.480281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.480260 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.529277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.529239 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhcz\" (UniqueName: \"kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.529463 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.529360 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.529463 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.529448 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.529592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.529475 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.529592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.529574 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.529710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.529650 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.581210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.581170 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"kube-root-ca.crt\"" Oct 31 01:12:17.581210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.581207 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"openshift-service-ca.crt\"" Oct 31 01:12:17.581421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.581224 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"simulator-w4-acd385-kserve-self-signed-certs\"" Oct 31 01:12:17.581421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.581270 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"default-dockercfg-pp82z\"" Oct 31 01:12:17.584218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.584198 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t"] Oct 31 01:12:17.631117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631086 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.631314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631125 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.631314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631152 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.631314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631188 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.631314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631243 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhcz\" (UniqueName: \"kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.631314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631301 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.631609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.631584 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.632045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.632026 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.632211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.632190 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.635054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.635034 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.636442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.636417 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.689856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.689818 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhcz\" (UniqueName: \"kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz\") pod \"simulator-w4-acd385-kserve-6698cbdc4f-xbg6t\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.796165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.796111 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:17.920970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.918933 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:17.920970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.919627 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:17.921494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:17.921126 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 01:12:18.191611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:18.191534 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t"] Oct 31 01:12:18.195093 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:12:18.195060 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a81b1d0_0c85_420f_ade0_2ace58b65dff.slice/crio-93a85aee710425ecca067b2c8b687449179c54bc844e3506d0a99c5c907e2561 WatchSource:0}: Error finding container 93a85aee710425ecca067b2c8b687449179c54bc844e3506d0a99c5c907e2561: Status 404 returned error can't find the container with id 93a85aee710425ecca067b2c8b687449179c54bc844e3506d0a99c5c907e2561 Oct 31 01:12:18.525283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:18.525206 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" event={"ID":"2a81b1d0-0c85-420f-ade0-2ace58b65dff","Type":"ContainerStarted","Data":"93a85aee710425ecca067b2c8b687449179c54bc844e3506d0a99c5c907e2561"} Oct 31 01:12:19.532474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:19.532443 5756 generic.go:356] "Generic (PLEG): container finished" podID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerID="d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872" exitCode=0 Oct 31 01:12:19.532971 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:19.532513 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" event={"ID":"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0","Type":"ContainerDied","Data":"d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872"} Oct 31 01:12:19.533121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:19.533104 5756 scope.go:122] "RemoveContainer" containerID="d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872" Oct 31 01:12:19.534269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:19.534244 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:12:19.534328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:19.534306 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:12:19.534368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:19.534345 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:12:20.566828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:20.566794 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" event={"ID":"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0","Type":"ContainerStarted","Data":"44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a"} Oct 31 01:12:20.570692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:20.568798 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" event={"ID":"2a81b1d0-0c85-420f-ade0-2ace58b65dff","Type":"ContainerStarted","Data":"4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f"} Oct 31 01:12:20.798997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:20.798966 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:20.807076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:20.806964 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podStartSLOduration=1.540397425 podStartE2EDuration="3.806948411s" totalImagesPullingTime="2.266550986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:20.80611205 +0000 UTC m=+912.613079585" watchObservedRunningTime="2026-10-31 01:12:20.806948411 +0000 UTC m=+912.613915955" Oct 31 01:12:21.315764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:21.315730 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:12:21.315959 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:21.315918 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 01:12:21.316022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:21.315969 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 01:12:22.315810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:22.315773 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 01:12:22.316240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:22.315828 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 01:12:23.315743 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:23.315708 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 01:12:23.315942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:23.315763 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 01:12:23.855465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:23.855435 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:12:24.315098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:24.315063 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 01:12:24.315272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:24.315117 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 01:12:25.315795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:25.315445 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 01:12:25.315795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:25.315501 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 01:12:26.319611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:26.319588 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:12:26.320035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:26.319856 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:12:26.320569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:26.320554 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:12:27.798548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:27.798513 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:27.811309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:27.811288 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:28.625167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:28.625140 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:12:32.066271 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:12:32.066234 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:12:34.002036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:34.002003 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:12:40.997510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:40.997474 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz"] Oct 31 01:12:41.003128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.003106 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.067195 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.067161 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t2p\" (UniqueName: \"kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.067364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.067283 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.067364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.067354 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.067481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.067410 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.067481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.067466 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.067566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.067508 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.106634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.106605 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"openshift-service-ca.crt\"" Oct 31 01:12:41.106634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.106608 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"kube-root-ca.crt\"" Oct 31 01:12:41.106827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.106674 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"default-dockercfg-jd84n\"" Oct 31 01:12:41.107034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.107019 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"premium-w0-26fbc4-kserve-self-signed-certs\"" Oct 31 01:12:41.108382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.108359 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz"] Oct 31 01:12:41.168568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.168538 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.168739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.168586 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.168739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.168607 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.168739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.168692 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t2p\" (UniqueName: \"kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.168919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.168751 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.168919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.168782 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.169045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.169024 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.169103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.169087 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.169182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.169161 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.170979 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.170956 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.171391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.171374 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.228024 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.227991 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t2p\" (UniqueName: \"kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p\") pod \"premium-w0-26fbc4-kserve-5d988688f9-l75bz\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.314655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.314609 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:41.710230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:41.710152 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz"] Oct 31 01:12:41.712654 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:12:41.712629 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod966522e7_caaf_4514_abd9_8b9111b971c0.slice/crio-35ce312eae781f51af40922f69cc5bf39ce18320c7503ff7dc056f589741b56d WatchSource:0}: Error finding container 35ce312eae781f51af40922f69cc5bf39ce18320c7503ff7dc056f589741b56d: Status 404 returned error can't find the container with id 35ce312eae781f51af40922f69cc5bf39ce18320c7503ff7dc056f589741b56d Oct 31 01:12:42.685682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:42.685652 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" event={"ID":"966522e7-caaf-4514-abd9-8b9111b971c0","Type":"ContainerStarted","Data":"a1634d2bf410afd1d6d7823998ccb3892b5b47185346eee721820e24154a0c7e"} Oct 31 01:12:42.686108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:42.685691 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" event={"ID":"966522e7-caaf-4514-abd9-8b9111b971c0","Type":"ContainerStarted","Data":"35ce312eae781f51af40922f69cc5bf39ce18320c7503ff7dc056f589741b56d"} Oct 31 01:12:42.800514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:42.800444 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podStartSLOduration=2.800431463 podStartE2EDuration="2.800431463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:42.79838421 +0000 UTC m=+934.605351754" watchObservedRunningTime="2026-10-31 01:12:42.800431463 +0000 UTC m=+934.607399005" Oct 31 01:12:43.315762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:43.315733 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:44.967192 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:44.967160 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:12:51.315750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:51.315712 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:51.327744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:51.327716 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:12:51.744912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:12:51.744816 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:13:01.489098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:01.489066 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:13:02.067337 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:13:02.067301 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:13:09.855581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.855545 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n"] Oct 31 01:13:09.859149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.859127 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:09.962821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.962786 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"kube-root-ca.crt\"" Oct 31 01:13:09.962821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.962787 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"openshift-service-ca.crt\"" Oct 31 01:13:09.963078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.962901 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"default-dockercfg-425j2\"" Oct 31 01:13:09.963078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.962913 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"simulator-w2-e621db-kserve-self-signed-certs\"" Oct 31 01:13:09.964543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.964517 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n"] Oct 31 01:13:09.979391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.979368 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:09.979486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.979396 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:09.979486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.979452 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:09.979571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.979531 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:09.979571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.979559 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzb98\" (UniqueName: \"kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:09.979651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:09.979584 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081010 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081067 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081100 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzb98\" (UniqueName: \"kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081128 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081184 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081368 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081463 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081489 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.081892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.081802 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.083545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.083525 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.083795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.083777 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.137842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.137747 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzb98\" (UniqueName: \"kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98\") pod \"simulator-w2-e621db-kserve-55667677cc-qrd9n\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.171016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.170982 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:10.569436 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.569303 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n"] Oct 31 01:13:10.571503 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:13:10.571477 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cad83b9_9b46_4174_b37b_7663d23b67e1.slice/crio-7a746cb542518da590619c08d463f03b4b47d9074ef4b14d26797b841e59107d WatchSource:0}: Error finding container 7a746cb542518da590619c08d463f03b4b47d9074ef4b14d26797b841e59107d: Status 404 returned error can't find the container with id 7a746cb542518da590619c08d463f03b4b47d9074ef4b14d26797b841e59107d Oct 31 01:13:10.831228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.831144 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" event={"ID":"4cad83b9-9b46-4174-b37b-7663d23b67e1","Type":"ContainerStarted","Data":"274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c"} Oct 31 01:13:10.831228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.831183 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" event={"ID":"4cad83b9-9b46-4174-b37b-7663d23b67e1","Type":"ContainerStarted","Data":"7a746cb542518da590619c08d463f03b4b47d9074ef4b14d26797b841e59107d"} Oct 31 01:13:10.951846 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:10.951789 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podStartSLOduration=1.951774704 podStartE2EDuration="1.951774704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:13:10.949843374 +0000 UTC m=+962.756810920" watchObservedRunningTime="2026-10-31 01:13:10.951774704 +0000 UTC m=+962.758742246" Oct 31 01:13:11.171449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:11.171377 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:17.052451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:17.052414 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:13:20.171279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:20.171243 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:20.183882 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:20.183840 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:20.903310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:20.903281 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:13:32.070076 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:13:32.070036 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:13:41.390399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:41.390366 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:13:55.458806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.458731 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh"] Oct 31 01:13:55.462254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.462239 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.553200 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.553159 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.553371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.553299 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.553421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.553389 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt8j\" (UniqueName: \"kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.563429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.563406 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:13:55.563429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.563411 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:13:55.565154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.565131 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh"] Oct 31 01:13:55.654203 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.654163 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.654395 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.654274 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt8j\" (UniqueName: \"kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.654395 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.654329 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.654973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.654950 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.656655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.656634 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.710991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.710920 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt8j\" (UniqueName: \"kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j\") pod \"maas-api-revoke-sub-aec6cf50-lgszh\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:55.774667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:55.774642 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:56.145988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:56.145923 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh"] Oct 31 01:13:56.148953 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:13:56.148925 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f5a13d_0a8e_4ef5_b5e0_04896ac0976f.slice/crio-9861a53ea159348503c0b1868612134f7584ea6b78d5add53a3cbf803b110c21 WatchSource:0}: Error finding container 9861a53ea159348503c0b1868612134f7584ea6b78d5add53a3cbf803b110c21: Status 404 returned error can't find the container with id 9861a53ea159348503c0b1868612134f7584ea6b78d5add53a3cbf803b110c21 Oct 31 01:13:56.922383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:56.922346 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:13:57.080481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:57.080452 5756 generic.go:356] "Generic (PLEG): container finished" podID="b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" containerID="6a4560e52603d459c7a270a3c8eecdf494d0d2fb826a8c77496674f74dd94d08" exitCode=0 Oct 31 01:13:57.080624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:57.080531 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" event={"ID":"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f","Type":"ContainerDied","Data":"6a4560e52603d459c7a270a3c8eecdf494d0d2fb826a8c77496674f74dd94d08"} Oct 31 01:13:57.080624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:57.080602 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" event={"ID":"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f","Type":"ContainerStarted","Data":"9861a53ea159348503c0b1868612134f7584ea6b78d5add53a3cbf803b110c21"} Oct 31 01:13:58.222271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.222246 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:58.384533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.384453 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca\") pod \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " Oct 31 01:13:58.384673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.384546 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j\" (UniqueName: \"kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j\") pod \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " Oct 31 01:13:58.384673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.384648 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca\") pod \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\" (UID: \"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f\") " Oct 31 01:13:58.384871 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.384836 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca" pod "b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" (UID: "b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:13:58.385077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.385060 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:58.386703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.386678 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca" pod "b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" (UID: "b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:13:58.386834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.386781 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j" pod "b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" (UID: "b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f"). InnerVolumeSpecName "kube-api-access-znt8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:13:58.486331 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.486299 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znt8j\" (UniqueName: \"kubernetes.io/projected/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-kube-api-access-znt8j\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:58.486508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:58.486390 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:13:59.092480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:59.092451 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" Oct 31 01:13:59.092637 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:59.092473 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh" event={"ID":"b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f","Type":"ContainerDied","Data":"9861a53ea159348503c0b1868612134f7584ea6b78d5add53a3cbf803b110c21"} Oct 31 01:13:59.092637 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:13:59.092537 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9861a53ea159348503c0b1868612134f7584ea6b78d5add53a3cbf803b110c21" Oct 31 01:14:02.066402 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:14:02.066364 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:14:13.176683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.176653 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp"] Oct 31 01:14:13.177133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.177102 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" containerName="revoke-keys" Oct 31 01:14:13.177133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.177115 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" containerName="revoke-keys" Oct 31 01:14:13.177205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.177180 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" containerName="revoke-keys" Oct 31 01:14:13.180303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.180285 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.231363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.231327 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmqf7\" (UniqueName: \"kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.231363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.231369 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.231590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.231392 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.233470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.233437 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:13.282893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.282846 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:14:13.284562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.284542 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp"] Oct 31 01:14:13.332739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.332711 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmqf7\" (UniqueName: \"kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.332928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.332756 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.332928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.332908 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.333504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.333483 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.335433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.335416 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.390220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.390186 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmqf7\" (UniqueName: \"kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7\") pod \"maas-api-revoke-sub-8af7a6df-7xlbp\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.493935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.493836 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:13.882718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:13.882695 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp"] Oct 31 01:14:13.885554 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:13.885523 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f53552_1f41_4aab_8543_591469dd9134.slice/crio-01f21922fdfbff9327f8349e6d38871bad8546b07d08b9494e9b90d6f28f2f1f WatchSource:0}: Error finding container 01f21922fdfbff9327f8349e6d38871bad8546b07d08b9494e9b90d6f28f2f1f: Status 404 returned error can't find the container with id 01f21922fdfbff9327f8349e6d38871bad8546b07d08b9494e9b90d6f28f2f1f Oct 31 01:14:14.172014 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:14.171980 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" event={"ID":"c8f53552-1f41-4aab-8543-591469dd9134","Type":"ContainerStarted","Data":"8999be19509a177460f683f88c5ee14f506799576bff5dc7c7801bdb227b2f86"} Oct 31 01:14:14.172159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:14.172046 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" event={"ID":"c8f53552-1f41-4aab-8543-591469dd9134","Type":"ContainerStarted","Data":"01f21922fdfbff9327f8349e6d38871bad8546b07d08b9494e9b90d6f28f2f1f"} Oct 31 01:14:14.286053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:14.286001 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" podStartSLOduration=1.285986339 podStartE2EDuration="1.285986339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:14.283903119 +0000 UTC m=+1026.090870664" watchObservedRunningTime="2026-10-31 01:14:14.285986339 +0000 UTC m=+1026.092953885" Oct 31 01:14:15.180277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:15.180190 5756 generic.go:356] "Generic (PLEG): container finished" podID="c8f53552-1f41-4aab-8543-591469dd9134" containerID="8999be19509a177460f683f88c5ee14f506799576bff5dc7c7801bdb227b2f86" exitCode=0 Oct 31 01:14:15.180413 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:15.180266 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" event={"ID":"c8f53552-1f41-4aab-8543-591469dd9134","Type":"ContainerDied","Data":"8999be19509a177460f683f88c5ee14f506799576bff5dc7c7801bdb227b2f86"} Oct 31 01:14:16.327568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.327543 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:16.367723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.367692 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca\") pod \"c8f53552-1f41-4aab-8543-591469dd9134\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " Oct 31 01:14:16.367856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.367792 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca\") pod \"c8f53552-1f41-4aab-8543-591469dd9134\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " Oct 31 01:14:16.367928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.367890 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7\" (UniqueName: \"kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7\") pod \"c8f53552-1f41-4aab-8543-591469dd9134\" (UID: \"c8f53552-1f41-4aab-8543-591469dd9134\") " Oct 31 01:14:16.368151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.368126 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca" pod "c8f53552-1f41-4aab-8543-591469dd9134" (UID: "c8f53552-1f41-4aab-8543-591469dd9134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:16.368345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.368321 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:16.370131 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.370105 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca" pod "c8f53552-1f41-4aab-8543-591469dd9134" (UID: "c8f53552-1f41-4aab-8543-591469dd9134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:16.370236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.370193 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7" pod "c8f53552-1f41-4aab-8543-591469dd9134" (UID: "c8f53552-1f41-4aab-8543-591469dd9134"). InnerVolumeSpecName "kube-api-access-pmqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:16.469714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.469627 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f53552-1f41-4aab-8543-591469dd9134-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:16.469714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:16.469667 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmqf7\" (UniqueName: \"kubernetes.io/projected/c8f53552-1f41-4aab-8543-591469dd9134-kube-api-access-pmqf7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:17.192426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:17.192398 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" Oct 31 01:14:17.192426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:17.192406 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp" event={"ID":"c8f53552-1f41-4aab-8543-591469dd9134","Type":"ContainerDied","Data":"01f21922fdfbff9327f8349e6d38871bad8546b07d08b9494e9b90d6f28f2f1f"} Oct 31 01:14:17.192632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:17.192449 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f21922fdfbff9327f8349e6d38871bad8546b07d08b9494e9b90d6f28f2f1f" Oct 31 01:14:19.896716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:19.896689 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7"] Oct 31 01:14:19.897126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:19.897112 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c8f53552-1f41-4aab-8543-591469dd9134" containerName="revoke-keys" Oct 31 01:14:19.897126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:19.897123 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f53552-1f41-4aab-8543-591469dd9134" containerName="revoke-keys" Oct 31 01:14:19.897243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:19.897233 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f53552-1f41-4aab-8543-591469dd9134" containerName="revoke-keys" Oct 31 01:14:19.900519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:19.900499 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.001340 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.001297 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"premium-w2-e621db-kserve-self-signed-certs\"" Oct 31 01:14:20.003051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.003030 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7"] Oct 31 01:14:20.006731 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.006710 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.006808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.006771 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmzj9\" (UniqueName: \"kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.006956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.006937 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.007003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.006972 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.007066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.007053 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.007130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.007078 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.107850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.107811 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmzj9\" (UniqueName: \"kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.107946 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108000 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108084 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108117 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108163 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108566 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108609 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.108722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.108653 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.110550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.110529 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.110645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.110628 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.166909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.166820 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmzj9\" (UniqueName: \"kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9\") pod \"premium-w2-e621db-kserve-5b9dcd847b-9htn7\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.212687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.212653 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:20.598425 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:20.598401 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7"] Oct 31 01:14:20.600502 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:20.600473 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7266da26_c8f4_429e_8295_8f25b463a6f1.slice/crio-e57e546b2843e177af239050bc5d350c10b10d7c60a724adbdfb613cea09b572 WatchSource:0}: Error finding container e57e546b2843e177af239050bc5d350c10b10d7c60a724adbdfb613cea09b572: Status 404 returned error can't find the container with id e57e546b2843e177af239050bc5d350c10b10d7c60a724adbdfb613cea09b572 Oct 31 01:14:21.216035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:21.216001 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" event={"ID":"7266da26-c8f4-429e-8295-8f25b463a6f1","Type":"ContainerStarted","Data":"320ff6ceda24321c65d26d97ee4bd193a3602669e90ba4cd6a41a19bcf052399"} Oct 31 01:14:21.216441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:21.216050 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" event={"ID":"7266da26-c8f4-429e-8295-8f25b463a6f1","Type":"ContainerStarted","Data":"e57e546b2843e177af239050bc5d350c10b10d7c60a724adbdfb613cea09b572"} Oct 31 01:14:21.350138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:21.350087 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podStartSLOduration=2.350072574 podStartE2EDuration="2.350072574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:21.34795748 +0000 UTC m=+1033.154925024" watchObservedRunningTime="2026-10-31 01:14:21.350072574 +0000 UTC m=+1033.157040148" Oct 31 01:14:22.213736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:22.213703 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:22.891412 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:22.891378 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb"] Oct 31 01:14:22.894920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:22.894898 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:22.997151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:22.997124 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:22.997310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:22.997153 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:14:22.999049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:22.999031 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb"] Oct 31 01:14:23.036407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.036374 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.036607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.036516 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77pj\" (UniqueName: \"kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.036607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.036587 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.137997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.137962 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77pj\" (UniqueName: \"kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.138181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.138012 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.138181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.138055 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.138683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.138660 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.140540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.140510 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.195548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.195473 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77pj\" (UniqueName: \"kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj\") pod \"maas-api-revoke-sub-484d0504-fzdsb\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.209190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.209160 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:23.582179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:23.582149 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb"] Oct 31 01:14:23.584129 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:23.584099 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e0545a_7218_4e90_8815_edd6ee0be6d7.slice/crio-251816f67db9e95f5f83063caff9a6b5dd1a47ef42db4942d4344c34436b5eb6 WatchSource:0}: Error finding container 251816f67db9e95f5f83063caff9a6b5dd1a47ef42db4942d4344c34436b5eb6: Status 404 returned error can't find the container with id 251816f67db9e95f5f83063caff9a6b5dd1a47ef42db4942d4344c34436b5eb6 Oct 31 01:14:24.234647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:24.234558 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" event={"ID":"88e0545a-7218-4e90-8815-edd6ee0be6d7","Type":"ContainerStarted","Data":"3c7c82e9830b42dfcae621602cefcd0b25505cee386109dbd738bbf448fd351a"} Oct 31 01:14:24.234647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:24.234606 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" event={"ID":"88e0545a-7218-4e90-8815-edd6ee0be6d7","Type":"ContainerStarted","Data":"251816f67db9e95f5f83063caff9a6b5dd1a47ef42db4942d4344c34436b5eb6"} Oct 31 01:14:24.348039 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:24.347917 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" podStartSLOduration=2.347898652 podStartE2EDuration="2.347898652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:24.34703923 +0000 UTC m=+1036.154006775" watchObservedRunningTime="2026-10-31 01:14:24.347898652 +0000 UTC m=+1036.154866195" Oct 31 01:14:25.241703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:25.241670 5756 generic.go:356] "Generic (PLEG): container finished" podID="88e0545a-7218-4e90-8815-edd6ee0be6d7" containerID="3c7c82e9830b42dfcae621602cefcd0b25505cee386109dbd738bbf448fd351a" exitCode=0 Oct 31 01:14:25.242128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:25.241767 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" event={"ID":"88e0545a-7218-4e90-8815-edd6ee0be6d7","Type":"ContainerDied","Data":"3c7c82e9830b42dfcae621602cefcd0b25505cee386109dbd738bbf448fd351a"} Oct 31 01:14:26.398634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.398599 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:26.475651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.475613 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj\" (UniqueName: \"kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj\") pod \"88e0545a-7218-4e90-8815-edd6ee0be6d7\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " Oct 31 01:14:26.475936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.475694 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca\") pod \"88e0545a-7218-4e90-8815-edd6ee0be6d7\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " Oct 31 01:14:26.475936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.475769 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca\") pod \"88e0545a-7218-4e90-8815-edd6ee0be6d7\" (UID: \"88e0545a-7218-4e90-8815-edd6ee0be6d7\") " Oct 31 01:14:26.476138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.476110 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca" pod "88e0545a-7218-4e90-8815-edd6ee0be6d7" (UID: "88e0545a-7218-4e90-8815-edd6ee0be6d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:26.476233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.476218 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.477934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.477907 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca" pod "88e0545a-7218-4e90-8815-edd6ee0be6d7" (UID: "88e0545a-7218-4e90-8815-edd6ee0be6d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:26.477934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.477924 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj" pod "88e0545a-7218-4e90-8815-edd6ee0be6d7" (UID: "88e0545a-7218-4e90-8815-edd6ee0be6d7"). InnerVolumeSpecName "kube-api-access-p77pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:26.577392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.577356 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88e0545a-7218-4e90-8815-edd6ee0be6d7-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.577392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.577387 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77pj\" (UniqueName: \"kubernetes.io/projected/88e0545a-7218-4e90-8815-edd6ee0be6d7-kube-api-access-p77pj\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:26.868561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:26.868484 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:14:27.258414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:27.258330 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" Oct 31 01:14:27.258826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:27.258352 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb" event={"ID":"88e0545a-7218-4e90-8815-edd6ee0be6d7","Type":"ContainerDied","Data":"251816f67db9e95f5f83063caff9a6b5dd1a47ef42db4942d4344c34436b5eb6"} Oct 31 01:14:27.258984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:27.258885 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="251816f67db9e95f5f83063caff9a6b5dd1a47ef42db4942d4344c34436b5eb6" Oct 31 01:14:29.348047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.348012 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9"] Oct 31 01:14:29.348538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.348478 5756 cpu_manager.go:414] "containerMap: removing container" podUID="88e0545a-7218-4e90-8815-edd6ee0be6d7" containerName="revoke-keys" Oct 31 01:14:29.348538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.348489 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e0545a-7218-4e90-8815-edd6ee0be6d7" containerName="revoke-keys" Oct 31 01:14:29.348652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.348567 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e0545a-7218-4e90-8815-edd6ee0be6d7" containerName="revoke-keys" Oct 31 01:14:29.351925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.351903 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.411700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.411670 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9"] Oct 31 01:14:29.458256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.458227 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:29.458256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.458237 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:14:29.510801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.510772 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.511016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.510815 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.511016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.510840 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjmt\" (UniqueName: \"kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.612303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.612212 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.612303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.612263 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.612303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.612283 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjmt\" (UniqueName: \"kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.612931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.612908 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.614924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.614901 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.670073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.670050 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjmt\" (UniqueName: \"kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt\") pod \"maas-api-revoke-sub-5125b284-82rn9\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:29.966851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:29.966767 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:30.213311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:30.213275 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:30.226250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:30.226184 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:30.286765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:30.286738 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:14:30.342626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:30.342599 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9"] Oct 31 01:14:30.346914 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:30.344665 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9543ac_e0c4_45f2_82e5_9b1d9e276483.slice/crio-201f123e67e7e7b0dc25aeda306fae336ef58f583440451af74031b0fe660ddc WatchSource:0}: Error finding container 201f123e67e7e7b0dc25aeda306fae336ef58f583440451af74031b0fe660ddc: Status 404 returned error can't find the container with id 201f123e67e7e7b0dc25aeda306fae336ef58f583440451af74031b0fe660ddc Oct 31 01:14:31.282358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:31.282325 5756 generic.go:356] "Generic (PLEG): container finished" podID="1a9543ac-e0c4-45f2-82e5-9b1d9e276483" containerID="9dd71d5c1e4997df8348e933404fa0bd977e8d92f3c42bf0b61ba414984d923c" exitCode=0 Oct 31 01:14:31.282738 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:31.282431 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" event={"ID":"1a9543ac-e0c4-45f2-82e5-9b1d9e276483","Type":"ContainerDied","Data":"9dd71d5c1e4997df8348e933404fa0bd977e8d92f3c42bf0b61ba414984d923c"} Oct 31 01:14:31.282738 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:31.282512 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" event={"ID":"1a9543ac-e0c4-45f2-82e5-9b1d9e276483","Type":"ContainerStarted","Data":"201f123e67e7e7b0dc25aeda306fae336ef58f583440451af74031b0fe660ddc"} Oct 31 01:14:32.065544 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:14:32.065501 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:14:32.428261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.428227 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:32.540076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.540038 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt\" (UniqueName: \"kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt\") pod \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " Oct 31 01:14:32.540276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.540124 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca\") pod \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " Oct 31 01:14:32.540276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.540212 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca\") pod \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\" (UID: \"1a9543ac-e0c4-45f2-82e5-9b1d9e276483\") " Oct 31 01:14:32.540599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.540572 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca" pod "1a9543ac-e0c4-45f2-82e5-9b1d9e276483" (UID: "1a9543ac-e0c4-45f2-82e5-9b1d9e276483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:32.542792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.542752 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca" pod "1a9543ac-e0c4-45f2-82e5-9b1d9e276483" (UID: "1a9543ac-e0c4-45f2-82e5-9b1d9e276483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:32.543102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.543078 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt" pod "1a9543ac-e0c4-45f2-82e5-9b1d9e276483" (UID: "1a9543ac-e0c4-45f2-82e5-9b1d9e276483"). InnerVolumeSpecName "kube-api-access-7kjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:32.641501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.641421 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:32.641501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.641449 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:32.641501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:32.641459 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjmt\" (UniqueName: \"kubernetes.io/projected/1a9543ac-e0c4-45f2-82e5-9b1d9e276483-kube-api-access-7kjmt\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:33.295146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:33.295118 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" Oct 31 01:14:33.295315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:33.295144 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9" event={"ID":"1a9543ac-e0c4-45f2-82e5-9b1d9e276483","Type":"ContainerDied","Data":"201f123e67e7e7b0dc25aeda306fae336ef58f583440451af74031b0fe660ddc"} Oct 31 01:14:33.295315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:33.295186 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201f123e67e7e7b0dc25aeda306fae336ef58f583440451af74031b0fe660ddc" Oct 31 01:14:38.239170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.239138 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf"] Oct 31 01:14:38.239610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.239595 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1a9543ac-e0c4-45f2-82e5-9b1d9e276483" containerName="revoke-keys" Oct 31 01:14:38.239649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.239611 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9543ac-e0c4-45f2-82e5-9b1d9e276483" containerName="revoke-keys" Oct 31 01:14:38.239690 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.239682 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9543ac-e0c4-45f2-82e5-9b1d9e276483" containerName="revoke-keys" Oct 31 01:14:38.242943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.242928 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.299280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.299242 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf"] Oct 31 01:14:38.345728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.345688 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:14:38.345728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.345692 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:38.400989 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.400954 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgv49\" (UniqueName: \"kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.401174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.401003 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.401174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.401024 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.502684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.502601 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgv49\" (UniqueName: \"kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.502684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.502650 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.502925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.502768 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.503274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.503244 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.505415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.505394 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.560135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.560096 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgv49\" (UniqueName: \"kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49\") pod \"maas-api-revoke-sub-33d7f4c3-mzfqf\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:38.855794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:38.855760 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:39.229073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:39.229046 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf"] Oct 31 01:14:39.231549 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:39.231519 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d52f99_7f8d_4965_b710_feb93f51668f.slice/crio-7d2952f35aa0a506004e1335a4a972c94807e1340ebc255c202a0bc274292708 WatchSource:0}: Error finding container 7d2952f35aa0a506004e1335a4a972c94807e1340ebc255c202a0bc274292708: Status 404 returned error can't find the container with id 7d2952f35aa0a506004e1335a4a972c94807e1340ebc255c202a0bc274292708 Oct 31 01:14:39.327920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:39.327885 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" event={"ID":"98d52f99-7f8d-4965-b710-feb93f51668f","Type":"ContainerStarted","Data":"7d2952f35aa0a506004e1335a4a972c94807e1340ebc255c202a0bc274292708"} Oct 31 01:14:40.334504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:40.334468 5756 generic.go:356] "Generic (PLEG): container finished" podID="98d52f99-7f8d-4965-b710-feb93f51668f" containerID="4ccc1e4375d7032c6e8410556e8908de3d7491afc39c126a1a55341798ae3859" exitCode=0 Oct 31 01:14:40.334922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:40.334519 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" event={"ID":"98d52f99-7f8d-4965-b710-feb93f51668f","Type":"ContainerDied","Data":"4ccc1e4375d7032c6e8410556e8908de3d7491afc39c126a1a55341798ae3859"} Oct 31 01:14:41.511168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.511140 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:41.529428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.529391 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca\") pod \"98d52f99-7f8d-4965-b710-feb93f51668f\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " Oct 31 01:14:41.529560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.529545 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49\" (UniqueName: \"kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49\") pod \"98d52f99-7f8d-4965-b710-feb93f51668f\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " Oct 31 01:14:41.529627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.529569 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca\") pod \"98d52f99-7f8d-4965-b710-feb93f51668f\" (UID: \"98d52f99-7f8d-4965-b710-feb93f51668f\") " Oct 31 01:14:41.529992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.529963 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca" pod "98d52f99-7f8d-4965-b710-feb93f51668f" (UID: "98d52f99-7f8d-4965-b710-feb93f51668f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:41.531812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.531789 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49" pod "98d52f99-7f8d-4965-b710-feb93f51668f" (UID: "98d52f99-7f8d-4965-b710-feb93f51668f"). InnerVolumeSpecName "kube-api-access-fgv49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:41.532143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.532116 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca" pod "98d52f99-7f8d-4965-b710-feb93f51668f" (UID: "98d52f99-7f8d-4965-b710-feb93f51668f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:41.631431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.631358 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:41.631431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.631384 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgv49\" (UniqueName: \"kubernetes.io/projected/98d52f99-7f8d-4965-b710-feb93f51668f-kube-api-access-fgv49\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:41.631431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:41.631396 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d52f99-7f8d-4965-b710-feb93f51668f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:42.345970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:42.345938 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" Oct 31 01:14:42.346122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:42.345967 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf" event={"ID":"98d52f99-7f8d-4965-b710-feb93f51668f","Type":"ContainerDied","Data":"7d2952f35aa0a506004e1335a4a972c94807e1340ebc255c202a0bc274292708"} Oct 31 01:14:42.346122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:42.346017 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2952f35aa0a506004e1335a4a972c94807e1340ebc255c202a0bc274292708" Oct 31 01:14:47.432629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:47.432594 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:14:49.812298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.812267 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff"] Oct 31 01:14:49.812711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.812689 5756 cpu_manager.go:414] "containerMap: removing container" podUID="98d52f99-7f8d-4965-b710-feb93f51668f" containerName="revoke-keys" Oct 31 01:14:49.812711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.812703 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d52f99-7f8d-4965-b710-feb93f51668f" containerName="revoke-keys" Oct 31 01:14:49.812807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.812798 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d52f99-7f8d-4965-b710-feb93f51668f" containerName="revoke-keys" Oct 31 01:14:49.816111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.816096 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:49.873660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.873631 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff"] Oct 31 01:14:49.919046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.919005 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:49.919213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.919051 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvwg\" (UniqueName: \"kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:49.919213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.919096 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:49.920326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.920291 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:14:49.920464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:49.920380 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:14:50.020376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.020346 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvwg\" (UniqueName: \"kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.020376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.020378 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.020630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.020500 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.021181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.021161 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.022960 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.022928 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.078344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.078267 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvwg\" (UniqueName: \"kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg\") pod \"maas-api-revoke-sub-32dfa249-4vzff\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.129352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.129316 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:50.498903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:50.498830 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff"] Oct 31 01:14:50.501313 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:50.501287 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd3b862_ba18_4a4b_94d7_08a2a1972a67.slice/crio-72cc55e81da3554bbcbc40b2d17b8df433db049a22f78064ed00e5eb642451f8 WatchSource:0}: Error finding container 72cc55e81da3554bbcbc40b2d17b8df433db049a22f78064ed00e5eb642451f8: Status 404 returned error can't find the container with id 72cc55e81da3554bbcbc40b2d17b8df433db049a22f78064ed00e5eb642451f8 Oct 31 01:14:51.399476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:51.399441 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" event={"ID":"1cd3b862-ba18-4a4b-94d7-08a2a1972a67","Type":"ContainerStarted","Data":"1db937c31b1523ad8d11a3eeb1baba228c9e7943a6354555e7ece69b7baf6341"} Oct 31 01:14:51.399805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:51.399503 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" event={"ID":"1cd3b862-ba18-4a4b-94d7-08a2a1972a67","Type":"ContainerStarted","Data":"72cc55e81da3554bbcbc40b2d17b8df433db049a22f78064ed00e5eb642451f8"} Oct 31 01:14:52.407122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.407086 5756 generic.go:356] "Generic (PLEG): container finished" podID="1cd3b862-ba18-4a4b-94d7-08a2a1972a67" containerID="1db937c31b1523ad8d11a3eeb1baba228c9e7943a6354555e7ece69b7baf6341" exitCode=0 Oct 31 01:14:52.407531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.407128 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" event={"ID":"1cd3b862-ba18-4a4b-94d7-08a2a1972a67","Type":"ContainerDied","Data":"1db937c31b1523ad8d11a3eeb1baba228c9e7943a6354555e7ece69b7baf6341"} Oct 31 01:14:52.541731 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.541702 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:52.649071 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.649041 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg\" (UniqueName: \"kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg\") pod \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " Oct 31 01:14:52.649252 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.649084 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca\") pod \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " Oct 31 01:14:52.649252 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.649131 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca\") pod \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\" (UID: \"1cd3b862-ba18-4a4b-94d7-08a2a1972a67\") " Oct 31 01:14:52.649445 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.649421 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca" pod "1cd3b862-ba18-4a4b-94d7-08a2a1972a67" (UID: "1cd3b862-ba18-4a4b-94d7-08a2a1972a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:14:52.649573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.649556 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.651218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.651197 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg" pod "1cd3b862-ba18-4a4b-94d7-08a2a1972a67" (UID: "1cd3b862-ba18-4a4b-94d7-08a2a1972a67"). InnerVolumeSpecName "kube-api-access-6kvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:14:52.651218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.651202 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca" pod "1cd3b862-ba18-4a4b-94d7-08a2a1972a67" (UID: "1cd3b862-ba18-4a4b-94d7-08a2a1972a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:14:52.751217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.751186 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvwg\" (UniqueName: \"kubernetes.io/projected/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-kube-api-access-6kvwg\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:52.751217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:52.751218 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd3b862-ba18-4a4b-94d7-08a2a1972a67-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:14:53.421545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:53.421508 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" Oct 31 01:14:53.421949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:53.421517 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff" event={"ID":"1cd3b862-ba18-4a4b-94d7-08a2a1972a67","Type":"ContainerDied","Data":"72cc55e81da3554bbcbc40b2d17b8df433db049a22f78064ed00e5eb642451f8"} Oct 31 01:14:53.421949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:53.421668 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72cc55e81da3554bbcbc40b2d17b8df433db049a22f78064ed00e5eb642451f8" Oct 31 01:14:55.571506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.571470 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56"] Oct 31 01:14:55.571935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.571922 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1cd3b862-ba18-4a4b-94d7-08a2a1972a67" containerName="revoke-keys" Oct 31 01:14:55.571977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.571936 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd3b862-ba18-4a4b-94d7-08a2a1972a67" containerName="revoke-keys" Oct 31 01:14:55.572037 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.572003 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd3b862-ba18-4a4b-94d7-08a2a1972a67" containerName="revoke-keys" Oct 31 01:14:55.575312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.575290 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.678716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.678673 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56"] Oct 31 01:14:55.681729 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.681706 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.681807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.681738 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.681807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.681764 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.681807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.681795 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.682023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.682000 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.682124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.682110 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.682183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.682155 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.682286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.682261 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.682324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.682306 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl8v8\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.783692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.783649 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.783930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.783710 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.783930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.783738 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl8v8\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.783923 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.783957 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.783997 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784054 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784087 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784113 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784203 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784114 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784409 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784496 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.784788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.784699 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.786493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.786471 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.786579 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.786473 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.840928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.840849 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.841031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.841010 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl8v8\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:55.888794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:55.888766 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:56.314045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.314014 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56"] Oct 31 01:14:56.317181 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:14:56.317153 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f0c5a5_4f65_4f6f_a0c4_93c655ee0d9f.slice/crio-bc7bafd12224cda21a3725ad537746c5db36b1d8c2e0eb27b574ddef66b56d00 WatchSource:0}: Error finding container bc7bafd12224cda21a3725ad537746c5db36b1d8c2e0eb27b574ddef66b56d00: Status 404 returned error can't find the container with id bc7bafd12224cda21a3725ad537746c5db36b1d8c2e0eb27b574ddef66b56d00 Oct 31 01:14:56.319401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.319371 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:14:56.319480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.319444 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:14:56.319480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.319473 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:14:56.440993 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.440949 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" event={"ID":"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f","Type":"ContainerStarted","Data":"36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102"} Oct 31 01:14:56.441187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.441016 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" event={"ID":"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f","Type":"ContainerStarted","Data":"bc7bafd12224cda21a3725ad537746c5db36b1d8c2e0eb27b574ddef66b56d00"} Oct 31 01:14:56.561364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.561299 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podStartSLOduration=1.561278638 podStartE2EDuration="1.561278638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:56.557913354 +0000 UTC m=+1068.364880897" watchObservedRunningTime="2026-10-31 01:14:56.561278638 +0000 UTC m=+1068.368246183" Oct 31 01:14:56.889583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:56.889557 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:14:57.891238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:57.891193 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 01:14:57.891238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:57.891247 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 01:14:58.889722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:58.889673 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 01:14:58.890020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:58.889758 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 01:14:59.890216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:59.890181 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 01:14:59.890610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:14:59.890235 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 01:15:00.179840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.179741 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps"] Oct 31 01:15:00.183702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.183675 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b"] Oct 31 01:15:00.183845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.183786 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.187277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.187255 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5"] Oct 31 01:15:00.187401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.187382 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.190795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.190775 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.248316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.248283 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:15:00.277247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.277214 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b"] Oct 31 01:15:00.299887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.299826 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps"] Oct 31 01:15:00.301693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.301665 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5"] Oct 31 01:15:00.358536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.358493 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz8m4\" (UniqueName: \"kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.358710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.358631 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx46f\" (UniqueName: \"kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.358710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.358668 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.358824 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.358801 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.358900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.358881 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.358983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.358970 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4smgb\" (UniqueName: \"kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.460574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.460493 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx46f\" (UniqueName: \"kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.460574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.460537 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.460800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.460611 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.460800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.460671 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.460800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.460730 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4smgb\" (UniqueName: \"kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.460800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.460783 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz8m4\" (UniqueName: \"kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.461411 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.461385 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.461498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.461452 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.461645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.461626 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.516999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.516971 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx46f\" (UniqueName: \"kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.517158 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.517001 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz8m4\" (UniqueName: \"kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.517158 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.516969 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4smgb\" (UniqueName: \"kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.798623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.798584 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:00.807037 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.807011 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:00.813244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.813217 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:00.892051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.891442 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 01:15:00.897357 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:00.897313 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 01:15:01.204601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.204577 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps"] Oct 31 01:15:01.206376 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:15:01.206346 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeef9075_9da5_4937_a28e_61944d0185cd.slice/crio-ffa7472bb5b2a300415c33b95d1e4fc1def3b020ca21aeeb2d986ce31d7c5a8a WatchSource:0}: Error finding container ffa7472bb5b2a300415c33b95d1e4fc1def3b020ca21aeeb2d986ce31d7c5a8a: Status 404 returned error can't find the container with id ffa7472bb5b2a300415c33b95d1e4fc1def3b020ca21aeeb2d986ce31d7c5a8a Oct 31 01:15:01.261207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.261184 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5"] Oct 31 01:15:01.262766 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:15:01.262744 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5be9bb_69b6_4e78_a1a9_e128809a2fdc.slice/crio-704eb545d2b1248d28c58c41908e6cafeceba9f2d6024dde907612e8cd0d87b3 WatchSource:0}: Error finding container 704eb545d2b1248d28c58c41908e6cafeceba9f2d6024dde907612e8cd0d87b3: Status 404 returned error can't find the container with id 704eb545d2b1248d28c58c41908e6cafeceba9f2d6024dde907612e8cd0d87b3 Oct 31 01:15:01.311503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.311475 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b"] Oct 31 01:15:01.314309 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:15:01.314285 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc220407_5df8_45c9_8036_280ca679e62d.slice/crio-bb4ceafc394edf0ef879dbca4e96f286d0089a609ccc5f0316bd3dbafe8ebf94 WatchSource:0}: Error finding container bb4ceafc394edf0ef879dbca4e96f286d0089a609ccc5f0316bd3dbafe8ebf94: Status 404 returned error can't find the container with id bb4ceafc394edf0ef879dbca4e96f286d0089a609ccc5f0316bd3dbafe8ebf94 Oct 31 01:15:01.474370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.474334 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" event={"ID":"beef9075-9da5-4937-a28e-61944d0185cd","Type":"ContainerStarted","Data":"ffa7472bb5b2a300415c33b95d1e4fc1def3b020ca21aeeb2d986ce31d7c5a8a"} Oct 31 01:15:01.475645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.475615 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" event={"ID":"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc","Type":"ContainerStarted","Data":"704eb545d2b1248d28c58c41908e6cafeceba9f2d6024dde907612e8cd0d87b3"} Oct 31 01:15:01.476806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.476781 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" event={"ID":"fc220407-5df8-45c9-8036-280ca679e62d","Type":"ContainerStarted","Data":"bb4ceafc394edf0ef879dbca4e96f286d0089a609ccc5f0316bd3dbafe8ebf94"} Oct 31 01:15:01.889768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.889732 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 01:15:01.889961 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:01.889787 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 01:15:02.078582 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:02.078546 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:15:02.485740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.485704 5756 generic.go:356] "Generic (PLEG): container finished" podID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerID="fa9f7d96099b503fae624ad78191264997a9fa1dc1242c2ecce529be8bffe598" exitCode=60 Oct 31 01:15:02.485959 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.485785 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" event={"ID":"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc","Type":"ContainerDied","Data":"fa9f7d96099b503fae624ad78191264997a9fa1dc1242c2ecce529be8bffe598"} Oct 31 01:15:02.486143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.486128 5756 scope.go:122] "RemoveContainer" containerID="fa9f7d96099b503fae624ad78191264997a9fa1dc1242c2ecce529be8bffe598" Oct 31 01:15:02.487983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.487955 5756 generic.go:356] "Generic (PLEG): container finished" podID="fc220407-5df8-45c9-8036-280ca679e62d" containerID="3048e1c4351b56074df11e90426af71c68979f695c0f367d0c36a722078b6473" exitCode=60 Oct 31 01:15:02.488099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.488038 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" event={"ID":"fc220407-5df8-45c9-8036-280ca679e62d","Type":"ContainerDied","Data":"3048e1c4351b56074df11e90426af71c68979f695c0f367d0c36a722078b6473"} Oct 31 01:15:02.488275 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.488248 5756 scope.go:122] "RemoveContainer" containerID="3048e1c4351b56074df11e90426af71c68979f695c0f367d0c36a722078b6473" Oct 31 01:15:02.490233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.490213 5756 generic.go:356] "Generic (PLEG): container finished" podID="beef9075-9da5-4937-a28e-61944d0185cd" containerID="473eeeb8975929d817d092e363d3730d0df8cc5fb2b228959016f216eeea2239" exitCode=60 Oct 31 01:15:02.490324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.490306 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" event={"ID":"beef9075-9da5-4937-a28e-61944d0185cd","Type":"ContainerDied","Data":"473eeeb8975929d817d092e363d3730d0df8cc5fb2b228959016f216eeea2239"} Oct 31 01:15:02.490558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:02.490542 5756 scope.go:122] "RemoveContainer" containerID="473eeeb8975929d817d092e363d3730d0df8cc5fb2b228959016f216eeea2239" Oct 31 01:15:03.018363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.018263 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:15:03.019043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.019021 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:15:03.020057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.020038 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:15:03.500663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.500629 5756 generic.go:356] "Generic (PLEG): container finished" podID="beef9075-9da5-4937-a28e-61944d0185cd" containerID="bb025e6219722ab23394831fe5318ae13f16f6844dcff7f5750c56f2436f8f74" exitCode=60 Oct 31 01:15:03.501143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.500718 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" event={"ID":"beef9075-9da5-4937-a28e-61944d0185cd","Type":"ContainerDied","Data":"bb025e6219722ab23394831fe5318ae13f16f6844dcff7f5750c56f2436f8f74"} Oct 31 01:15:03.501143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.500778 5756 scope.go:122] "RemoveContainer" containerID="473eeeb8975929d817d092e363d3730d0df8cc5fb2b228959016f216eeea2239" Oct 31 01:15:03.501143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.501024 5756 scope.go:122] "RemoveContainer" containerID="bb025e6219722ab23394831fe5318ae13f16f6844dcff7f5750c56f2436f8f74" Oct 31 01:15:03.501271 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:03.501253 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps_odh-ai-gateway-infra(beef9075-9da5-4937-a28e-61944d0185cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" podUID="beef9075-9da5-4937-a28e-61944d0185cd" Oct 31 01:15:03.503083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.503060 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" event={"ID":"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc","Type":"ContainerStarted","Data":"dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459"} Oct 31 01:15:03.504849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.504826 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" event={"ID":"fc220407-5df8-45c9-8036-280ca679e62d","Type":"ContainerStarted","Data":"14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539"} Oct 31 01:15:03.733152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.733051 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" podStartSLOduration=3.733036142 podStartE2EDuration="3.733036142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:03.731545426 +0000 UTC m=+1075.538512972" watchObservedRunningTime="2026-10-31 01:15:03.733036142 +0000 UTC m=+1075.540003684" Oct 31 01:15:03.846939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:03.846871 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" podStartSLOduration=3.846842925 podStartE2EDuration="3.846842925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:03.844780752 +0000 UTC m=+1075.651748294" watchObservedRunningTime="2026-10-31 01:15:03.846842925 +0000 UTC m=+1075.653810467" Oct 31 01:15:04.512582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.512555 5756 scope.go:122] "RemoveContainer" containerID="bb025e6219722ab23394831fe5318ae13f16f6844dcff7f5750c56f2436f8f74" Oct 31 01:15:04.513055 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:04.512807 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps_odh-ai-gateway-infra(beef9075-9da5-4937-a28e-61944d0185cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" podUID="beef9075-9da5-4937-a28e-61944d0185cd" Oct 31 01:15:04.514288 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.514261 5756 generic.go:356] "Generic (PLEG): container finished" podID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerID="dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459" exitCode=60 Oct 31 01:15:04.514399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.514351 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" event={"ID":"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc","Type":"ContainerDied","Data":"dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459"} Oct 31 01:15:04.514452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.514407 5756 scope.go:122] "RemoveContainer" containerID="fa9f7d96099b503fae624ad78191264997a9fa1dc1242c2ecce529be8bffe598" Oct 31 01:15:04.514525 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.514509 5756 scope.go:122] "RemoveContainer" containerID="dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459" Oct 31 01:15:04.514750 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:04.514726 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5_odh-ai-gateway-infra(fb5be9bb-69b6-4e78-a1a9-e128809a2fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" Oct 31 01:15:04.516179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.516157 5756 generic.go:356] "Generic (PLEG): container finished" podID="fc220407-5df8-45c9-8036-280ca679e62d" containerID="14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539" exitCode=60 Oct 31 01:15:04.516274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.516224 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" event={"ID":"fc220407-5df8-45c9-8036-280ca679e62d","Type":"ContainerDied","Data":"14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539"} Oct 31 01:15:04.516382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.516370 5756 scope.go:122] "RemoveContainer" containerID="14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539" Oct 31 01:15:04.516576 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:04.516561 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b_odh-ai-gateway-infra(fc220407-5df8-45c9-8036-280ca679e62d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" podUID="fc220407-5df8-45c9-8036-280ca679e62d" Oct 31 01:15:04.526595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:04.526578 5756 scope.go:122] "RemoveContainer" containerID="3048e1c4351b56074df11e90426af71c68979f695c0f367d0c36a722078b6473" Oct 31 01:15:05.524293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:05.524254 5756 scope.go:122] "RemoveContainer" containerID="dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459" Oct 31 01:15:05.524688 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:05.524500 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5_odh-ai-gateway-infra(fb5be9bb-69b6-4e78-a1a9-e128809a2fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" Oct 31 01:15:05.525800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:05.525782 5756 scope.go:122] "RemoveContainer" containerID="14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539" Oct 31 01:15:05.526008 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:05.525991 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b_odh-ai-gateway-infra(fc220407-5df8-45c9-8036-280ca679e62d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" podUID="fc220407-5df8-45c9-8036-280ca679e62d" Oct 31 01:15:09.077660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:09.077622 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:15:10.502525 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.502495 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz"] Oct 31 01:15:10.507784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.507766 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.561497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.561461 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5cb\" (UniqueName: \"kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.561681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.561593 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.562777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.561738 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.563073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.562856 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz"] Oct 31 01:15:10.609140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.609113 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:10.663944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.663911 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.664149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.664085 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5cb\" (UniqueName: \"kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.664149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.664118 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.664746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.664722 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.666615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.666595 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.720618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.720585 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5cb\" (UniqueName: \"kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb\") pod \"maas-api-revoke-sub-8b07904f-h9cjz\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:10.820562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:10.820531 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:11.192436 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:11.192410 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz"] Oct 31 01:15:11.194294 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:15:11.194263 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebb645d_1c2b_4fd0_aad7_925c81c806c4.slice/crio-78cb70ee8ecad3bb9577781ea714b6cb3f5525994c1a374e32a508835990ed43 WatchSource:0}: Error finding container 78cb70ee8ecad3bb9577781ea714b6cb3f5525994c1a374e32a508835990ed43: Status 404 returned error can't find the container with id 78cb70ee8ecad3bb9577781ea714b6cb3f5525994c1a374e32a508835990ed43 Oct 31 01:15:11.564594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:11.564558 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" event={"ID":"9ebb645d-1c2b-4fd0-aad7-925c81c806c4","Type":"ContainerStarted","Data":"287b30c4440393661f12bf99582c34af5c5cccb364fcfbb88106b8068da18af4"} Oct 31 01:15:11.564999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:11.564612 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" event={"ID":"9ebb645d-1c2b-4fd0-aad7-925c81c806c4","Type":"ContainerStarted","Data":"78cb70ee8ecad3bb9577781ea714b6cb3f5525994c1a374e32a508835990ed43"} Oct 31 01:15:11.725988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:11.725932 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" podStartSLOduration=1.725920024 podStartE2EDuration="1.725920024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:11.677086449 +0000 UTC m=+1083.484053994" watchObservedRunningTime="2026-10-31 01:15:11.725920024 +0000 UTC m=+1083.532887566" Oct 31 01:15:12.571361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:12.571326 5756 generic.go:356] "Generic (PLEG): container finished" podID="9ebb645d-1c2b-4fd0-aad7-925c81c806c4" containerID="287b30c4440393661f12bf99582c34af5c5cccb364fcfbb88106b8068da18af4" exitCode=0 Oct 31 01:15:12.571738 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:12.571413 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" event={"ID":"9ebb645d-1c2b-4fd0-aad7-925c81c806c4","Type":"ContainerDied","Data":"287b30c4440393661f12bf99582c34af5c5cccb364fcfbb88106b8068da18af4"} Oct 31 01:15:13.716445 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.716421 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:13.794847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.794816 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca\") pod \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " Oct 31 01:15:13.795044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.794887 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb\" (UniqueName: \"kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb\") pod \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " Oct 31 01:15:13.795044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.794927 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca\") pod \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\" (UID: \"9ebb645d-1c2b-4fd0-aad7-925c81c806c4\") " Oct 31 01:15:13.795241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.795216 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca" pod "9ebb645d-1c2b-4fd0-aad7-925c81c806c4" (UID: "9ebb645d-1c2b-4fd0-aad7-925c81c806c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:13.797118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.797088 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca" pod "9ebb645d-1c2b-4fd0-aad7-925c81c806c4" (UID: "9ebb645d-1c2b-4fd0-aad7-925c81c806c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:13.797224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.797117 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb" pod "9ebb645d-1c2b-4fd0-aad7-925c81c806c4" (UID: "9ebb645d-1c2b-4fd0-aad7-925c81c806c4"). InnerVolumeSpecName "kube-api-access-4r5cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:13.896794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.896719 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:13.896794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.896746 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r5cb\" (UniqueName: \"kubernetes.io/projected/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-kube-api-access-4r5cb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:13.896794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:13.896756 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ebb645d-1c2b-4fd0-aad7-925c81c806c4-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:14.304803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:14.304774 5756 scope.go:122] "RemoveContainer" containerID="14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539" Oct 31 01:15:14.584632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:14.584555 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" Oct 31 01:15:14.584632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:14.584590 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz" event={"ID":"9ebb645d-1c2b-4fd0-aad7-925c81c806c4","Type":"ContainerDied","Data":"78cb70ee8ecad3bb9577781ea714b6cb3f5525994c1a374e32a508835990ed43"} Oct 31 01:15:14.584803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:14.584634 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78cb70ee8ecad3bb9577781ea714b6cb3f5525994c1a374e32a508835990ed43" Oct 31 01:15:15.304781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:15.304753 5756 scope.go:122] "RemoveContainer" containerID="dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459" Oct 31 01:15:15.591938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:15.591909 5756 generic.go:356] "Generic (PLEG): container finished" podID="fc220407-5df8-45c9-8036-280ca679e62d" containerID="4f26d1e557364736f4990fb7f969e85f2b30ecee6c91109447d30d48b2cb5a55" exitCode=60 Oct 31 01:15:15.592112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:15.591956 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" event={"ID":"fc220407-5df8-45c9-8036-280ca679e62d","Type":"ContainerDied","Data":"4f26d1e557364736f4990fb7f969e85f2b30ecee6c91109447d30d48b2cb5a55"} Oct 31 01:15:15.592112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:15.592025 5756 scope.go:122] "RemoveContainer" containerID="14b0dae0d42b2e1f02e66b4cbbd7eec856fffb1e0f06476abaeaef9121a78539" Oct 31 01:15:15.592405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:15.592385 5756 scope.go:122] "RemoveContainer" containerID="4f26d1e557364736f4990fb7f969e85f2b30ecee6c91109447d30d48b2cb5a55" Oct 31 01:15:15.592672 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:15.592653 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b_odh-ai-gateway-infra(fc220407-5df8-45c9-8036-280ca679e62d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" podUID="fc220407-5df8-45c9-8036-280ca679e62d" Oct 31 01:15:16.601490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.601453 5756 generic.go:356] "Generic (PLEG): container finished" podID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerID="a752f23625512c0b88e1ff9828a2f7ecc1e0569f07a5174ccf101cacd4a42038" exitCode=60 Oct 31 01:15:16.601490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.601494 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" event={"ID":"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc","Type":"ContainerDied","Data":"a752f23625512c0b88e1ff9828a2f7ecc1e0569f07a5174ccf101cacd4a42038"} Oct 31 01:15:16.602076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.601537 5756 scope.go:122] "RemoveContainer" containerID="dc7f94936f5b01f748aa815fa6e3bf8d398d08bbe55615d1c1d4647fd83b5459" Oct 31 01:15:16.602076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.601934 5756 scope.go:122] "RemoveContainer" containerID="a752f23625512c0b88e1ff9828a2f7ecc1e0569f07a5174ccf101cacd4a42038" Oct 31 01:15:16.602240 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:16.602219 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5_odh-ai-gateway-infra(fb5be9bb-69b6-4e78-a1a9-e128809a2fdc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" Oct 31 01:15:16.718264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.718226 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b"] Oct 31 01:15:16.860743 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.860683 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:16.928827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.928781 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca\") pod \"fc220407-5df8-45c9-8036-280ca679e62d\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " Oct 31 01:15:16.929035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.928920 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4\" (UniqueName: \"kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4\") pod \"fc220407-5df8-45c9-8036-280ca679e62d\" (UID: \"fc220407-5df8-45c9-8036-280ca679e62d\") " Oct 31 01:15:16.929146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.929124 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca" pod "fc220407-5df8-45c9-8036-280ca679e62d" (UID: "fc220407-5df8-45c9-8036-280ca679e62d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:16.929360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.929343 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc220407-5df8-45c9-8036-280ca679e62d-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:16.931040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:16.931016 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4" pod "fc220407-5df8-45c9-8036-280ca679e62d" (UID: "fc220407-5df8-45c9-8036-280ca679e62d"). InnerVolumeSpecName "kube-api-access-qz8m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:17.030764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.030733 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz8m4\" (UniqueName: \"kubernetes.io/projected/fc220407-5df8-45c9-8036-280ca679e62d-kube-api-access-qz8m4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:17.610436 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.610398 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" event={"ID":"fc220407-5df8-45c9-8036-280ca679e62d","Type":"ContainerDied","Data":"bb4ceafc394edf0ef879dbca4e96f286d0089a609ccc5f0316bd3dbafe8ebf94"} Oct 31 01:15:17.610436 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.610417 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b" Oct 31 01:15:17.610979 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.610468 5756 scope.go:122] "RemoveContainer" containerID="4f26d1e557364736f4990fb7f969e85f2b30ecee6c91109447d30d48b2cb5a55" Oct 31 01:15:17.729754 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.729714 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5"] Oct 31 01:15:17.773214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.773182 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b"] Oct 31 01:15:17.783934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.783894 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890155-c7p4b"] Oct 31 01:15:17.878736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.878674 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:17.940623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.940589 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca\") pod \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " Oct 31 01:15:17.940767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.940641 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f\" (UniqueName: \"kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f\") pod \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\" (UID: \"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc\") " Oct 31 01:15:17.941063 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.941039 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca" pod "fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" (UID: "fb5be9bb-69b6-4e78-a1a9-e128809a2fdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:17.942988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:17.942965 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f" pod "fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" (UID: "fb5be9bb-69b6-4e78-a1a9-e128809a2fdc"). InnerVolumeSpecName "kube-api-access-gx46f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:18.042235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.042202 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.042235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.042230 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx46f\" (UniqueName: \"kubernetes.io/projected/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc-kube-api-access-gx46f\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:18.308053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.308022 5756 scope.go:122] "RemoveContainer" containerID="bb025e6219722ab23394831fe5318ae13f16f6844dcff7f5750c56f2436f8f74" Oct 31 01:15:18.311298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.311278 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc220407-5df8-45c9-8036-280ca679e62d" path="/var/lib/kubelet/pods/fc220407-5df8-45c9-8036-280ca679e62d/volumes" Oct 31 01:15:18.617653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.617578 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" Oct 31 01:15:18.618144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.617575 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5" event={"ID":"fb5be9bb-69b6-4e78-a1a9-e128809a2fdc","Type":"ContainerDied","Data":"704eb545d2b1248d28c58c41908e6cafeceba9f2d6024dde907612e8cd0d87b3"} Oct 31 01:15:18.618144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.617736 5756 scope.go:122] "RemoveContainer" containerID="a752f23625512c0b88e1ff9828a2f7ecc1e0569f07a5174ccf101cacd4a42038" Oct 31 01:15:18.778567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.778538 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5"] Oct 31 01:15:18.784470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:18.784434 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890155-m48s5"] Oct 31 01:15:19.626224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:19.626188 5756 generic.go:356] "Generic (PLEG): container finished" podID="beef9075-9da5-4937-a28e-61944d0185cd" containerID="5c999a44907f6f3bfbaa58e9a46c609c840900811798da2615d965a81139348e" exitCode=60 Oct 31 01:15:19.626664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:19.626258 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" event={"ID":"beef9075-9da5-4937-a28e-61944d0185cd","Type":"ContainerDied","Data":"5c999a44907f6f3bfbaa58e9a46c609c840900811798da2615d965a81139348e"} Oct 31 01:15:19.626664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:19.626318 5756 scope.go:122] "RemoveContainer" containerID="bb025e6219722ab23394831fe5318ae13f16f6844dcff7f5750c56f2436f8f74" Oct 31 01:15:19.626664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:19.626629 5756 scope.go:122] "RemoveContainer" containerID="5c999a44907f6f3bfbaa58e9a46c609c840900811798da2615d965a81139348e" Oct 31 01:15:19.626947 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:19.626926 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps_odh-ai-gateway-infra(beef9075-9da5-4937-a28e-61944d0185cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" podUID="beef9075-9da5-4937-a28e-61944d0185cd" Oct 31 01:15:20.310800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.310770 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" path="/var/lib/kubelet/pods/fb5be9bb-69b6-4e78-a1a9-e128809a2fdc/volumes" Oct 31 01:15:20.748207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.748133 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps"] Oct 31 01:15:20.894928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.894904 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:20.973637 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.973605 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb\" (UniqueName: \"kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb\") pod \"beef9075-9da5-4937-a28e-61944d0185cd\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " Oct 31 01:15:20.973804 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.973666 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca\") pod \"beef9075-9da5-4937-a28e-61944d0185cd\" (UID: \"beef9075-9da5-4937-a28e-61944d0185cd\") " Oct 31 01:15:20.974087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.974064 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca" pod "beef9075-9da5-4937-a28e-61944d0185cd" (UID: "beef9075-9da5-4937-a28e-61944d0185cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:20.975935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:20.975914 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb" pod "beef9075-9da5-4937-a28e-61944d0185cd" (UID: "beef9075-9da5-4937-a28e-61944d0185cd"). InnerVolumeSpecName "kube-api-access-4smgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:21.074904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.074843 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4smgb\" (UniqueName: \"kubernetes.io/projected/beef9075-9da5-4937-a28e-61944d0185cd-kube-api-access-4smgb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:21.074904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.074902 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beef9075-9da5-4937-a28e-61944d0185cd-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:21.641485 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.641455 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" Oct 31 01:15:21.641681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.641497 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps" event={"ID":"beef9075-9da5-4937-a28e-61944d0185cd","Type":"ContainerDied","Data":"ffa7472bb5b2a300415c33b95d1e4fc1def3b020ca21aeeb2d986ce31d7c5a8a"} Oct 31 01:15:21.641681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.641563 5756 scope.go:122] "RemoveContainer" containerID="5c999a44907f6f3bfbaa58e9a46c609c840900811798da2615d965a81139348e" Oct 31 01:15:21.803526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.803495 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps"] Oct 31 01:15:21.807558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:21.807532 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890155-9jgps"] Oct 31 01:15:22.311073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:22.311040 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beef9075-9da5-4937-a28e-61944d0185cd" path="/var/lib/kubelet/pods/beef9075-9da5-4937-a28e-61944d0185cd/volumes" Oct 31 01:15:32.066910 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:15:32.066842 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:15:32.781701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.781666 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c"] Oct 31 01:15:32.782149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782135 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782151 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782165 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782170 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782177 5756 cpu_manager.go:414] "containerMap: removing container" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782182 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782192 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:32.782198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782196 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782202 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782206 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782238 5756 cpu_manager.go:414] "containerMap: removing container" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782242 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782247 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9ebb645d-1c2b-4fd0-aad7-925c81c806c4" containerName="revoke-keys" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782251 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebb645d-1c2b-4fd0-aad7-925c81c806c4" containerName="revoke-keys" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782306 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782314 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782321 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782326 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782331 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782337 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782348 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:32.782399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.782352 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebb645d-1c2b-4fd0-aad7-925c81c806c4" containerName="revoke-keys" Oct 31 01:15:32.787016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.786999 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:32.841467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.841439 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c"] Oct 31 01:15:32.888263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.888236 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:15:32.888263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.888251 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:15:32.905528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.905494 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:32.905682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.905632 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:32.905759 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:32.905719 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllns\" (UniqueName: \"kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.007229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.007198 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.007402 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.007265 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllns\" (UniqueName: \"kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.007402 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.007346 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.008086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.008059 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.009954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.009932 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.064599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.064527 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllns\" (UniqueName: \"kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns\") pod \"maas-api-revoke-sub-300e228a-6nf8c\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.100295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.100266 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:33.473367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.473343 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c"] Oct 31 01:15:33.475502 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:15:33.475473 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13991d9_47b7_4847_8638_6640dcef60d0.slice/crio-74046bcfb53a42a705faf665828add93e64bc9175d31b7f12a97fb1a8f9ed43b WatchSource:0}: Error finding container 74046bcfb53a42a705faf665828add93e64bc9175d31b7f12a97fb1a8f9ed43b: Status 404 returned error can't find the container with id 74046bcfb53a42a705faf665828add93e64bc9175d31b7f12a97fb1a8f9ed43b Oct 31 01:15:33.708978 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:33.708940 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" event={"ID":"a13991d9-47b7-4847-8638-6640dcef60d0","Type":"ContainerStarted","Data":"74046bcfb53a42a705faf665828add93e64bc9175d31b7f12a97fb1a8f9ed43b"} Oct 31 01:15:34.716751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:34.716718 5756 generic.go:356] "Generic (PLEG): container finished" podID="a13991d9-47b7-4847-8638-6640dcef60d0" containerID="cd8efeb6155dddb16e03c6c50325b47e725219b5f955c9f278978dd1fb546071" exitCode=0 Oct 31 01:15:34.717168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:34.716787 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" event={"ID":"a13991d9-47b7-4847-8638-6640dcef60d0","Type":"ContainerDied","Data":"cd8efeb6155dddb16e03c6c50325b47e725219b5f955c9f278978dd1fb546071"} Oct 31 01:15:35.918502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:35.918478 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:36.039963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.039930 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns\" (UniqueName: \"kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns\") pod \"a13991d9-47b7-4847-8638-6640dcef60d0\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " Oct 31 01:15:36.040319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.040293 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca\") pod \"a13991d9-47b7-4847-8638-6640dcef60d0\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " Oct 31 01:15:36.040446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.040345 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca\") pod \"a13991d9-47b7-4847-8638-6640dcef60d0\" (UID: \"a13991d9-47b7-4847-8638-6640dcef60d0\") " Oct 31 01:15:36.040655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.040633 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca" pod "a13991d9-47b7-4847-8638-6640dcef60d0" (UID: "a13991d9-47b7-4847-8638-6640dcef60d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:15:36.040765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.040750 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:36.042317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.042295 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns" pod "a13991d9-47b7-4847-8638-6640dcef60d0" (UID: "a13991d9-47b7-4847-8638-6640dcef60d0"). InnerVolumeSpecName "kube-api-access-fllns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:15:36.042703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.042688 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca" pod "a13991d9-47b7-4847-8638-6640dcef60d0" (UID: "a13991d9-47b7-4847-8638-6640dcef60d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:15:36.142493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.142458 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13991d9-47b7-4847-8638-6640dcef60d0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:36.142493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.142494 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fllns\" (UniqueName: \"kubernetes.io/projected/a13991d9-47b7-4847-8638-6640dcef60d0-kube-api-access-fllns\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:15:36.692945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.690216 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:15:36.739312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.739277 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" event={"ID":"a13991d9-47b7-4847-8638-6640dcef60d0","Type":"ContainerDied","Data":"74046bcfb53a42a705faf665828add93e64bc9175d31b7f12a97fb1a8f9ed43b"} Oct 31 01:15:36.739312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.739303 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c" Oct 31 01:15:36.739520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:36.739330 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74046bcfb53a42a705faf665828add93e64bc9175d31b7f12a97fb1a8f9ed43b" Oct 31 01:15:42.171705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.171669 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns"] Oct 31 01:15:42.172184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172169 5756 cpu_manager.go:414] "containerMap: removing container" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:42.172242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172185 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:42.172242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172200 5756 cpu_manager.go:414] "containerMap: removing container" podUID="a13991d9-47b7-4847-8638-6640dcef60d0" containerName="revoke-keys" Oct 31 01:15:42.172242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172206 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13991d9-47b7-4847-8638-6640dcef60d0" containerName="revoke-keys" Oct 31 01:15:42.172242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172231 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:42.172242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172239 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc220407-5df8-45c9-8036-280ca679e62d" containerName="cleanup" Oct 31 01:15:42.172391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172254 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:42.172391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172260 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:42.172446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172416 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="beef9075-9da5-4937-a28e-61944d0185cd" containerName="cleanup" Oct 31 01:15:42.172446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172430 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13991d9-47b7-4847-8638-6640dcef60d0" containerName="revoke-keys" Oct 31 01:15:42.172446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.172442 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5be9bb-69b6-4e78-a1a9-e128809a2fdc" containerName="cleanup" Oct 31 01:15:42.177081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.177063 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.229023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.228992 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ec695b-openshift-default-dockercfg-ksrrv\"" Oct 31 01:15:42.280617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.280588 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns"] Oct 31 01:15:42.321026 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.320994 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321042 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321122 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321179 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321204 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321229 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321304 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321320 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcxh\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.321458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.321363 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.422532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422459 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.422532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422493 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.422532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422515 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.422795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422640 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.422795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422693 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.422795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422737 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422814 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422843 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcxh\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422879 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422890 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422903 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.422954 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.423140 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.423507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.423488 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.424874 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.424836 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.425224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.425204 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.479099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.479071 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.479352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.479329 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcxh\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.489896 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.489875 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:42.913917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.913881 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns"] Oct 31 01:15:42.953783 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:15:42.953753 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f82f9f2_6da1_495b_98e5_9c00182e6691.slice/crio-676ecbfed86af80d5ed837a28de3c3461fa5ef57991aea6260c13f8ecd93d8bc WatchSource:0}: Error finding container 676ecbfed86af80d5ed837a28de3c3461fa5ef57991aea6260c13f8ecd93d8bc: Status 404 returned error can't find the container with id 676ecbfed86af80d5ed837a28de3c3461fa5ef57991aea6260c13f8ecd93d8bc Oct 31 01:15:42.955841 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.955808 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:15:42.955927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.955895 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:15:42.955974 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:42.955924 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:15:43.782925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:43.782887 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" event={"ID":"2f82f9f2-6da1-495b-98e5-9c00182e6691","Type":"ContainerStarted","Data":"33753ee81b2c1ae9a559e5b728c6bce279773d515e659593d890e49ab1b4bff6"} Oct 31 01:15:43.783435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:43.782950 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" event={"ID":"2f82f9f2-6da1-495b-98e5-9c00182e6691","Type":"ContainerStarted","Data":"676ecbfed86af80d5ed837a28de3c3461fa5ef57991aea6260c13f8ecd93d8bc"} Oct 31 01:15:44.005638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:44.005589 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podStartSLOduration=2.005573778 podStartE2EDuration="2.005573778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:15:43.954786535 +0000 UTC m=+1115.761754079" watchObservedRunningTime="2026-10-31 01:15:44.005573778 +0000 UTC m=+1115.812541320" Oct 31 01:15:44.490513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:44.490471 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:44.492044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:44.492010 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 01:15:44.492163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:44.492068 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 01:15:45.490283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:45.490252 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 01:15:45.490668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:45.490303 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 01:15:46.490320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:46.490287 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 01:15:46.490685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:46.490344 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 01:15:47.490325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:47.490292 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 01:15:47.490728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:47.490341 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 01:15:48.490404 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:48.490370 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 01:15:48.490833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:48.490437 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 01:15:49.490477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:49.490436 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 01:15:49.490916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:49.490521 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 01:15:50.495373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:50.495344 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:50.495895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:50.495629 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:15:50.496358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:15:50.496343 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:16:02.066830 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:16:02.066789 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:16:04.831469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:04.831439 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:16:28.619783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:28.619748 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:16:32.066698 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:16:32.066661 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:16:32.448367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.448284 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf"] Oct 31 01:16:32.452438 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.452416 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.463097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.463077 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv"] Oct 31 01:16:32.466503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.466488 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.547246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.547202 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf"] Oct 31 01:16:32.570277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570243 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570279 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570417 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570448 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bv4\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570542 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570478 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570542 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570518 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99dxv\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570587 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570648 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570673 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570696 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570717 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.570810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570800 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570821 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570847 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.570970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570950 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.571010 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.570968 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.571047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.571014 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.571047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.571036 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.591404 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.591374 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv"] Oct 31 01:16:32.672458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672417 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672475 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672526 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672578 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.672635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672606 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672638 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672656 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672673 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672722 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672749 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672779 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.672822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672812 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672855 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672917 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.672981 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673008 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673041 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bv4\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673043 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673065 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673103 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99dxv\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673119 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.673674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673222 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.673674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673299 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.673674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673456 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.673674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673491 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.674062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673937 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.674062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.673951 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.674062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.674029 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.675532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.675511 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.676051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.675978 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.676202 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.676153 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.676946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.676927 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.730799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.730717 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.734707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.734673 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bv4\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4\") pod \"maas-default-gateway-openshift-default-fc8c56547-79dvv\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:32.734832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.734686 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99dxv\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.734832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.734814 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token\") pod \"maas-default-gateway-openshift-default-fc8c56547-zzqhf\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.766596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.766566 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:32.779018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:32.778993 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:33.199534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.199499 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf"] Oct 31 01:16:33.201687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.201663 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv"] Oct 31 01:16:33.202837 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:16:33.202810 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52e8f18_4093_4a3e_8d8f_a502b1adee86.slice/crio-228f074d4abc5731027e57edb3e69d4643b8ac7f82b317e8a3a7b736faa137a3 WatchSource:0}: Error finding container 228f074d4abc5731027e57edb3e69d4643b8ac7f82b317e8a3a7b736faa137a3: Status 404 returned error can't find the container with id 228f074d4abc5731027e57edb3e69d4643b8ac7f82b317e8a3a7b736faa137a3 Oct 31 01:16:33.203416 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:16:33.203393 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad7eb15_197a_441b_aaa2_df5bc76d84a6.slice/crio-50d3fb359788af4294e6ca6cd4a1b9614577573133d950b4f6d9e26c81177bcd WatchSource:0}: Error finding container 50d3fb359788af4294e6ca6cd4a1b9614577573133d950b4f6d9e26c81177bcd: Status 404 returned error can't find the container with id 50d3fb359788af4294e6ca6cd4a1b9614577573133d950b4f6d9e26c81177bcd Oct 31 01:16:33.205158 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.205121 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:33.205223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.205199 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:33.205263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.205233 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:33.205655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.205589 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:33.205655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.205645 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:33.205730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:33.205675 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:34.104318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.104281 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" event={"ID":"cad7eb15-197a-441b-aaa2-df5bc76d84a6","Type":"ContainerStarted","Data":"63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf"} Oct 31 01:16:34.104509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.104346 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" event={"ID":"cad7eb15-197a-441b-aaa2-df5bc76d84a6","Type":"ContainerStarted","Data":"50d3fb359788af4294e6ca6cd4a1b9614577573133d950b4f6d9e26c81177bcd"} Oct 31 01:16:34.106016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.105984 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" event={"ID":"c52e8f18-4093-4a3e-8d8f-a502b1adee86","Type":"ContainerStarted","Data":"16ab930dc3504ed07bfaa7fec293927c66292018ef325e3ebca026cbc26da43c"} Oct 31 01:16:34.106128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.106038 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" event={"ID":"c52e8f18-4093-4a3e-8d8f-a502b1adee86","Type":"ContainerStarted","Data":"228f074d4abc5731027e57edb3e69d4643b8ac7f82b317e8a3a7b736faa137a3"} Oct 31 01:16:34.331890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.329174 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podStartSLOduration=2.329156027 podStartE2EDuration="2.329156027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:34.278504063 +0000 UTC m=+1166.085471607" watchObservedRunningTime="2026-10-31 01:16:34.329156027 +0000 UTC m=+1166.136123572" Oct 31 01:16:34.397939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.397789 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podStartSLOduration=2.39777519 podStartE2EDuration="2.39777519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:34.397033959 +0000 UTC m=+1166.204001516" watchObservedRunningTime="2026-10-31 01:16:34.39777519 +0000 UTC m=+1166.204742732" Oct 31 01:16:34.767942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.767249 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:34.772891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.771170 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:34.772891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.771229 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:34.780269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.780239 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:34.784926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.784228 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:34.784926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:34.784303 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:35.778097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:35.776301 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:35.778097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:35.776379 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:35.781155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:35.781127 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:35.781285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:35.781171 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:36.770289 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:36.770252 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:36.770673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:36.770642 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:36.783179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:36.783148 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:36.783724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:36.783696 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:37.769888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:37.769292 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:37.769888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:37.769357 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:37.780562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:37.780397 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:37.780562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:37.780456 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:38.767354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:38.767317 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:38.768088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:38.768051 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:38.779969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:38.779593 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:38.779969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:38.779653 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:39.767840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:39.767803 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:39.768332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:39.767857 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:39.780248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:39.780216 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:39.780368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:39.780269 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:40.767383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:40.767347 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:16:40.767586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:40.767403 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 01:16:40.779847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:40.779809 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:16:40.780337 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:40.779895 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 01:16:41.789233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:41.789198 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:41.789933 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:41.789912 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:41.790611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:41.790588 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:16:41.874697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:41.874664 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:41.875020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:41.875002 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:41.875844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:41.875830 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:16:42.618996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.618960 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6"] Oct 31 01:16:42.623378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.623352 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.683843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.683812 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6"] Oct 31 01:16:42.732917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.732874 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.732962 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733024 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733164 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733213 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733308 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733294 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733361 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733420 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733410 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbq74\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.733454 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.733443 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835061 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835101 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835121 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835159 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835182 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835204 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbq74\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835237 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835338 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835364 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.835565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835542 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.836042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835687 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.836042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835755 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.836042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.835751 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.836164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.836146 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.837557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.837533 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.837906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.837889 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.891443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.891367 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbq74\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.891443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.891421 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:42.938655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:42.938622 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:43.369687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:43.369658 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6"] Oct 31 01:16:43.408121 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:16:43.408092 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f2323d_4169_4e62_a62d_a8a3e2d1f595.slice/crio-af354c40af44f6d2c821a018467f600a88afdbe6781387288b5d875f1f881b2d WatchSource:0}: Error finding container af354c40af44f6d2c821a018467f600a88afdbe6781387288b5d875f1f881b2d: Status 404 returned error can't find the container with id af354c40af44f6d2c821a018467f600a88afdbe6781387288b5d875f1f881b2d Oct 31 01:16:43.410521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:43.410492 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:43.410603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:43.410552 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:43.410603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:43.410583 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:44.218877 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:44.218825 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" event={"ID":"21f2323d-4169-4e62-a62d-a8a3e2d1f595","Type":"ContainerStarted","Data":"a0e04691ff32760ff39e36164d1cd729eb85e1d8a88074ccf412eb62cbb328e3"} Oct 31 01:16:44.219292 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:44.218903 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" event={"ID":"21f2323d-4169-4e62-a62d-a8a3e2d1f595","Type":"ContainerStarted","Data":"af354c40af44f6d2c821a018467f600a88afdbe6781387288b5d875f1f881b2d"} Oct 31 01:16:44.385712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:44.385658 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podStartSLOduration=2.385640986 podStartE2EDuration="2.385640986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:44.382948049 +0000 UTC m=+1176.189915605" watchObservedRunningTime="2026-10-31 01:16:44.385640986 +0000 UTC m=+1176.192608598" Oct 31 01:16:44.944641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:44.944467 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:44.954055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:44.954021 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 01:16:44.954234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:44.954076 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 01:16:45.940356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:45.940271 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 01:16:45.941079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:45.941051 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 01:16:46.946423 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:46.946389 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 01:16:46.947046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:46.947014 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 01:16:47.953265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:47.953227 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 01:16:47.953688 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:47.953283 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 01:16:48.942889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:48.941223 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 01:16:48.942889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:48.941316 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 01:16:49.939797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:49.939761 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 31 01:16:49.940213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:49.939813 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 31 01:16:50.943633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:50.943602 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:50.944077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:50.943897 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:50.944667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:50.944651 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:16:52.877751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:52.877719 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:16:55.222253 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.222219 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp"] Oct 31 01:16:55.226210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.226190 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.274607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.274573 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.274757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.274645 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.274757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.274730 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.274889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.274759 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.274889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.274814 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.274986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.274915 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9qr\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.275020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.275008 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.275053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.275040 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.275095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.275063 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.277902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.277832 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-97b317-openshift-default-dockercfg-4lml6\"" Oct 31 01:16:55.330300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.330269 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp"] Oct 31 01:16:55.376638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.376593 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.376813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.376671 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9qr\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.376813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.376729 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.376813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.376774 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.376813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.376802 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.376915 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377015 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377121 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377017 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377175 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377181 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377520 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.377569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.377563 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.378068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.378045 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.379762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.379740 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.380123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.380084 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.432946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.432924 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.433079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.433003 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9qr\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.539686 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.539654 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:55.965250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:55.965172 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp"] Oct 31 01:16:56.005737 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:16:56.005709 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1451e36d_b999_42b2_a6ea_21bdf88e6798.slice/crio-51a7bacd868a79aa053d8f79c881158921e4810dd0585025ea0f7dd194deea86 WatchSource:0}: Error finding container 51a7bacd868a79aa053d8f79c881158921e4810dd0585025ea0f7dd194deea86: Status 404 returned error can't find the container with id 51a7bacd868a79aa053d8f79c881158921e4810dd0585025ea0f7dd194deea86 Oct 31 01:16:56.008117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.008085 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:56.008226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.008154 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:56.008226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.008182 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:16:56.316225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.316187 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" event={"ID":"1451e36d-b999-42b2-a6ea-21bdf88e6798","Type":"ContainerStarted","Data":"904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6"} Oct 31 01:16:56.316684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.316258 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" event={"ID":"1451e36d-b999-42b2-a6ea-21bdf88e6798","Type":"ContainerStarted","Data":"51a7bacd868a79aa053d8f79c881158921e4810dd0585025ea0f7dd194deea86"} Oct 31 01:16:56.435657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.435605 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podStartSLOduration=1.435590718 podStartE2EDuration="1.435590718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:16:56.432739895 +0000 UTC m=+1188.239707453" watchObservedRunningTime="2026-10-31 01:16:56.435590718 +0000 UTC m=+1188.242558304" Oct 31 01:16:56.540888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:56.540035 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:16:57.552891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:57.550236 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:16:57.552891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:57.550315 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 01:16:58.541394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:58.541355 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:16:58.541657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:58.541631 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 01:16:59.555047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:59.555007 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:16:59.555453 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:16:59.555091 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 01:17:00.540295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:00.540254 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:17:00.540573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:00.540544 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 01:17:01.542256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:01.542221 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:17:01.542844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:01.542816 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 01:17:02.067802 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:17:02.067766 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:17:02.540800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.540763 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:17:02.541031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.540816 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 31 01:17:02.590223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.590189 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8"] Oct 31 01:17:02.594193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.594169 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.652512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.652482 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8"] Oct 31 01:17:02.685318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.685286 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.685523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.685362 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.685523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.685469 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.685523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.685516 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.685687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.685562 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8f56\" (UniqueName: \"kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.685741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.685708 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.697752 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.697727 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"distinct-w2-e621db-kserve-self-signed-certs\"" Oct 31 01:17:02.786980 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.786938 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.787811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.787714 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.787989 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.787957 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.788085 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.788052 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.788148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.788083 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.788148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.788114 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.788243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.788160 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8f56\" (UniqueName: \"kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.788708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.788685 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.792682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.789209 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.792885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.792147 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.794581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.794560 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.846467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.846433 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8f56\" (UniqueName: \"kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56\") pod \"distinct-w2-e621db-kserve-6fddbd456c-w7gf8\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:02.909930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:02.908536 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:03.380299 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:03.380269 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8"] Oct 31 01:17:03.382665 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:17:03.382640 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod398da614_85b5_47d4_8f6f_7f13bf3b145f.slice/crio-e65938e8e1d980748fa28799bf55fa25ada053fd8310d0c58b0037bde29f5fce WatchSource:0}: Error finding container e65938e8e1d980748fa28799bf55fa25ada053fd8310d0c58b0037bde29f5fce: Status 404 returned error can't find the container with id e65938e8e1d980748fa28799bf55fa25ada053fd8310d0c58b0037bde29f5fce Oct 31 01:17:03.545019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:03.544990 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:17:03.545456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:03.545432 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:17:03.546177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:03.546157 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:17:04.390054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:04.390014 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" event={"ID":"398da614-85b5-47d4-8f6f-7f13bf3b145f","Type":"ContainerStarted","Data":"ecb4de90141f9d246aeadc803f430103513483b996d0256b2b28ee392bd743fd"} Oct 31 01:17:04.390482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:04.390062 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" event={"ID":"398da614-85b5-47d4-8f6f-7f13bf3b145f","Type":"ContainerStarted","Data":"e65938e8e1d980748fa28799bf55fa25ada053fd8310d0c58b0037bde29f5fce"} Oct 31 01:17:04.504243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:04.504182 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podStartSLOduration=2.504168328 podStartE2EDuration="2.504168328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:04.502439039 +0000 UTC m=+1196.309406585" watchObservedRunningTime="2026-10-31 01:17:04.504168328 +0000 UTC m=+1196.311135870" Oct 31 01:17:04.909799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:04.909771 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:08.482085 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:08.482054 5756 scope.go:122] "RemoveContainer" containerID="1fa63275a0ebf252f61cd498b3402f860f283e0a559bb110ec0918dabf2381f5" Oct 31 01:17:08.492278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:08.492254 5756 scope.go:122] "RemoveContainer" containerID="f52be11ded565899d82eb7c8e42fb6b33bab2924ff7df11ac8ccfcaa05b1a03a" Oct 31 01:17:08.500967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:08.500938 5756 scope.go:122] "RemoveContainer" containerID="2bd781ccb2531bd1b82e8003503bb917e059540366b931f6a49f490e9e6338cf" Oct 31 01:17:08.509235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:08.509220 5756 scope.go:122] "RemoveContainer" containerID="f373b4d2ceaf7ff7335f37b9be84c08b63eb5f9d5ecb7ed5744b9fe3e4ae12fd" Oct 31 01:17:12.422655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:12.422625 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:17:12.916328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:12.916297 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:12.945628 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:12.945597 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:13.463717 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:13.463685 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:17:32.065855 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:17:32.065814 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:17:33.299159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:33.299129 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:17:51.333309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.333278 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v"] Oct 31 01:17:51.337386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.337366 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.411439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.411406 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v"] Oct 31 01:17:51.440067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.440042 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-acd385\"/\"distinct-w4-acd385-kserve-self-signed-certs\"" Oct 31 01:17:51.501429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.501396 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.501638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.501465 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljzz\" (UniqueName: \"kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.501638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.501561 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.501638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.501587 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.501812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.501645 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.501812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.501695 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.602812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.602732 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.602812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.602799 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.602847 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.603022 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljzz\" (UniqueName: \"kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.603177 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.603216 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.603275 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.603322 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.603501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.603459 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.605335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.605314 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.605920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.605904 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.659402 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.659361 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljzz\" (UniqueName: \"kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz\") pod \"distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:51.949609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:51.949525 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:17:52.322935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:52.322915 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v"] Oct 31 01:17:52.324584 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:17:52.324554 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebbdc099_30e9_48db_a755_05ebedb07f49.slice/crio-49852ad6ea90a9c10bd78c2179f056610007809c3761e29aba2afe96b30ccd23 WatchSource:0}: Error finding container 49852ad6ea90a9c10bd78c2179f056610007809c3761e29aba2afe96b30ccd23: Status 404 returned error can't find the container with id 49852ad6ea90a9c10bd78c2179f056610007809c3761e29aba2afe96b30ccd23 Oct 31 01:17:52.681007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:52.680920 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" event={"ID":"ebbdc099-30e9-48db-a755-05ebedb07f49","Type":"ContainerStarted","Data":"9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e"} Oct 31 01:17:52.681007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:52.680964 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" event={"ID":"ebbdc099-30e9-48db-a755-05ebedb07f49","Type":"ContainerStarted","Data":"49852ad6ea90a9c10bd78c2179f056610007809c3761e29aba2afe96b30ccd23"} Oct 31 01:17:52.794154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:52.794104 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podStartSLOduration=1.794089882 podStartE2EDuration="1.794089882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:52.79314896 +0000 UTC m=+1244.600116495" watchObservedRunningTime="2026-10-31 01:17:52.794089882 +0000 UTC m=+1244.601057425" Oct 31 01:17:52.950063 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:17:52.949988 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:18:01.950608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:01.950575 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:18:01.969788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:01.969751 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:18:02.065850 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:18:02.065805 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:18:02.753561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:02.753537 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:18:04.968848 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:04.968815 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:18:08.545462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:08.545432 5756 scope.go:122] "RemoveContainer" containerID="cbe9e9f7b4a1912d233d45ed2966717ba42ae65ae538cdf0893e2818f6153afa" Oct 31 01:18:27.160995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.160908 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r"] Oct 31 01:18:27.165050 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.165033 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.234565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234527 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.234735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234572 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.234735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234658 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.234833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234736 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.234833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234799 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.234833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234819 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zvg\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.235009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234840 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.235009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.234977 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.235091 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.235034 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.335777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335734 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.335777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335778 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335826 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335854 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335896 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zvg\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335925 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.335989 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336074 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336133 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336271 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336464 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336507 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336574 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.336785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.336764 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.338353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.338331 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.338772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.338752 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.364398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.364362 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r"] Oct 31 01:18:27.392249 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.392212 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.392395 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.392262 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zvg\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg\") pod \"e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.478002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.477926 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:27.932127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.932099 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r"] Oct 31 01:18:27.933597 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:18:27.933570 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3955cd72_831e_4360_a2b7_bef5dd3f36ea.slice/crio-168e6ef640c6ff8fcc100a72b526f0ac880243cb280b77107e47c99a0729b190 WatchSource:0}: Error finding container 168e6ef640c6ff8fcc100a72b526f0ac880243cb280b77107e47c99a0729b190: Status 404 returned error can't find the container with id 168e6ef640c6ff8fcc100a72b526f0ac880243cb280b77107e47c99a0729b190 Oct 31 01:18:27.935989 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.935955 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:18:27.936101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.936026 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:18:27.936101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:27.936056 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:18:28.910321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:28.910293 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" event={"ID":"3955cd72-831e-4360-a2b7-bef5dd3f36ea","Type":"ContainerStarted","Data":"dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a"} Oct 31 01:18:28.910718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:28.910338 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" event={"ID":"3955cd72-831e-4360-a2b7-bef5dd3f36ea","Type":"ContainerStarted","Data":"168e6ef640c6ff8fcc100a72b526f0ac880243cb280b77107e47c99a0729b190"} Oct 31 01:18:29.125257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:29.125197 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podStartSLOduration=2.125182528 podStartE2EDuration="2.125182528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:29.07479993 +0000 UTC m=+1280.881767498" watchObservedRunningTime="2026-10-31 01:18:29.125182528 +0000 UTC m=+1280.932150071" Oct 31 01:18:29.478115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:29.478083 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:29.480006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:29.479979 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:18:29.480178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:29.480038 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 01:18:30.478490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:30.478455 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:18:30.478894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:30.478507 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 01:18:31.472363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.472331 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj"] Oct 31 01:18:31.476715 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.476563 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.479178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.479154 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:18:31.479469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.479204 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 01:18:31.494199 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.494174 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvb4v\" (UniqueName: \"kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.494353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.494206 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.494353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.494233 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.494489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.494358 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.494489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.494407 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.494588 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.494490 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.595610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.595580 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvb4v\" (UniqueName: \"kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.595610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.595614 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.595833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.595635 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.595833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.595712 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.595833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.595771 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.595833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.595815 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.596145 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.596122 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.596205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.596150 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.596250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.596203 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.597985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.597969 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.599714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.599690 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e621db\"/\"distinct-2-w2-e621db-kserve-self-signed-certs\"" Oct 31 01:18:31.602231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.602209 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj"] Oct 31 01:18:31.609694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.609665 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.664620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.664550 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvb4v\" (UniqueName: \"kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v\") pod \"distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:31.790018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:31.789990 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:32.082501 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:18:32.082423 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:18:32.244885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:32.242464 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj"] Oct 31 01:18:32.262794 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:18:32.262762 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode89f39e4_29e9_4374_b0ac_c88ae0836486.slice/crio-e066d1e9797a967844b41cbdc6e36a30521054330c2c09c72fdf0632209caa7c WatchSource:0}: Error finding container e066d1e9797a967844b41cbdc6e36a30521054330c2c09c72fdf0632209caa7c: Status 404 returned error can't find the container with id e066d1e9797a967844b41cbdc6e36a30521054330c2c09c72fdf0632209caa7c Oct 31 01:18:32.478936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:32.478837 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:18:32.478936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:32.478914 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 01:18:33.018205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:33.018171 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" event={"ID":"e89f39e4-29e9-4374-b0ac-c88ae0836486","Type":"ContainerStarted","Data":"2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987"} Oct 31 01:18:33.018611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:33.018216 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" event={"ID":"e89f39e4-29e9-4374-b0ac-c88ae0836486","Type":"ContainerStarted","Data":"e066d1e9797a967844b41cbdc6e36a30521054330c2c09c72fdf0632209caa7c"} Oct 31 01:18:33.478474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:33.478439 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:18:33.478680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:33.478498 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 01:18:33.790208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:33.790176 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:34.478812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:34.478774 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 31 01:18:34.479367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:34.478844 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 31 01:18:35.483150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:35.483119 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:35.483516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:35.483388 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:35.484050 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:35.484037 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:18:35.698092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:35.698030 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podStartSLOduration=4.698010225 podStartE2EDuration="4.698010225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:18:33.145176978 +0000 UTC m=+1284.952144536" watchObservedRunningTime="2026-10-31 01:18:35.698010225 +0000 UTC m=+1287.504977769" Oct 31 01:18:37.257399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:37.257361 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:18:41.790926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:41.790893 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:41.803393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:41.803368 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:42.101015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:42.100934 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:18:58.653581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.653545 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm"] Oct 31 01:18:58.657479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.657452 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.761879 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.761823 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm"] Oct 31 01:18:58.797758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.797716 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.797987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.797796 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.797987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.797823 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.797987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.797842 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.797987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.797955 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffwtn\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.798238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.797990 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.798238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.798034 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.798238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.798067 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.798238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.798092 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.899623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.899562 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffwtn\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.900474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.900447 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.901798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.901762 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.902002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.901983 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.902326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.902309 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.902711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.902695 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.903556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.903539 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.904480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.904419 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.905072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.905043 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.905218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.905200 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.905451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.904293 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.905547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.903122 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.905633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.903409 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.907904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.902638 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.909087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.909059 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.910135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.910118 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.956652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.956622 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffwtn\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.966711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.966679 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token\") pod \"e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:58.971642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:58.971614 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:18:59.067892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:59.067649 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh"] Oct 31 01:18:59.071952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:59.071894 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aec6cf50-lgszh"] Oct 31 01:18:59.412474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:59.412434 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm"] Oct 31 01:18:59.415478 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:18:59.415449 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice/crio-b75d9f3b4b0ec83234b51547e74734046802f7e0b632a1e98ccf389a1f54b123 WatchSource:0}: Error finding container b75d9f3b4b0ec83234b51547e74734046802f7e0b632a1e98ccf389a1f54b123: Status 404 returned error can't find the container with id b75d9f3b4b0ec83234b51547e74734046802f7e0b632a1e98ccf389a1f54b123 Oct 31 01:18:59.417983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:59.417949 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:18:59.418095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:59.418036 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:18:59.418095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:18:59.418071 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:00.274886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.274749 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" event={"ID":"2087da4a-c455-430b-81cc-7c291549f3ab","Type":"ContainerStarted","Data":"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539"} Oct 31 01:19:00.274886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.274822 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" event={"ID":"2087da4a-c455-430b-81cc-7c291549f3ab","Type":"ContainerStarted","Data":"b75d9f3b4b0ec83234b51547e74734046802f7e0b632a1e98ccf389a1f54b123"} Oct 31 01:19:00.351887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.351733 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f" path="/var/lib/kubelet/pods/b0f5a13d-0a8e-4ef5-b5e0-04896ac0976f/volumes" Oct 31 01:19:00.404887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.404626 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podStartSLOduration=2.404604728 podStartE2EDuration="2.404604728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:00.403370987 +0000 UTC m=+1312.210338532" watchObservedRunningTime="2026-10-31 01:19:00.404604728 +0000 UTC m=+1312.211572272" Oct 31 01:19:00.979888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.979071 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:19:00.979888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.979211 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 01:19:00.979888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:00.979259 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 01:19:01.977368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:01.977330 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 01:19:01.978048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:01.978013 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 01:19:02.101359 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:19:02.101306 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:19:02.972412 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:02.972377 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 01:19:02.972577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:02.972440 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 01:19:03.972773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:03.972743 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 01:19:03.973254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:03.972798 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 01:19:04.972787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:04.972754 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 01:19:04.973299 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:04.972814 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 01:19:05.972621 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:05.972586 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 01:19:05.972792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:05.972642 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 01:19:06.977279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:06.977248 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:19:06.977684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:06.977520 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:19:06.978299 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:06.978284 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:19:07.619502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.619474 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4"] Oct 31 01:19:07.623722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.623705 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.676897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.676662 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:07.677066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.676969 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:07.680132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.680109 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4"] Oct 31 01:19:07.715548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.715517 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.715685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.715632 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfqj\" (UniqueName: \"kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.715874 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.715837 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.816744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.816710 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.816941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.816770 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfqj\" (UniqueName: \"kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.816941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.816929 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.817649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.817623 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.819274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.819257 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.874593 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.874530 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfqj\" (UniqueName: \"kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj\") pod \"maas-api-revoke-sub-8979b5c9-pcqt4\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:07.937651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:07.937621 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:08.321580 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:08.321548 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c143d7_3e4a_4b79_8080_58ed1909aab6.slice/crio-a5fa29dd54a46e22f1b7a10b1522eb957a87c4e1e4fb8a92b6e3a1c14d52ac9a WatchSource:0}: Error finding container a5fa29dd54a46e22f1b7a10b1522eb957a87c4e1e4fb8a92b6e3a1c14d52ac9a: Status 404 returned error can't find the container with id a5fa29dd54a46e22f1b7a10b1522eb957a87c4e1e4fb8a92b6e3a1c14d52ac9a Oct 31 01:19:08.322030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:08.321959 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4"] Oct 31 01:19:08.355069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:08.355037 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" event={"ID":"60c143d7-3e4a-4b79-8080-58ed1909aab6","Type":"ContainerStarted","Data":"a5fa29dd54a46e22f1b7a10b1522eb957a87c4e1e4fb8a92b6e3a1c14d52ac9a"} Oct 31 01:19:09.362605 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:09.362570 5756 generic.go:356] "Generic (PLEG): container finished" podID="60c143d7-3e4a-4b79-8080-58ed1909aab6" containerID="2841a5956c41eb49b326dcfff4d59e7270e624d6f23e466bffea48cc52fe089e" exitCode=0 Oct 31 01:19:09.363052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:09.362623 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" event={"ID":"60c143d7-3e4a-4b79-8080-58ed1909aab6","Type":"ContainerDied","Data":"2841a5956c41eb49b326dcfff4d59e7270e624d6f23e466bffea48cc52fe089e"} Oct 31 01:19:10.518396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.518374 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:10.548117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.548093 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca\") pod \"60c143d7-3e4a-4b79-8080-58ed1909aab6\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " Oct 31 01:19:10.548240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.548127 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj\" (UniqueName: \"kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj\") pod \"60c143d7-3e4a-4b79-8080-58ed1909aab6\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " Oct 31 01:19:10.548282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.548243 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca\") pod \"60c143d7-3e4a-4b79-8080-58ed1909aab6\" (UID: \"60c143d7-3e4a-4b79-8080-58ed1909aab6\") " Oct 31 01:19:10.548578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.548554 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca" pod "60c143d7-3e4a-4b79-8080-58ed1909aab6" (UID: "60c143d7-3e4a-4b79-8080-58ed1909aab6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:10.549079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.549055 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.550371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.550346 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca" pod "60c143d7-3e4a-4b79-8080-58ed1909aab6" (UID: "60c143d7-3e4a-4b79-8080-58ed1909aab6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:10.550565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.550545 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj" pod "60c143d7-3e4a-4b79-8080-58ed1909aab6" (UID: "60c143d7-3e4a-4b79-8080-58ed1909aab6"). InnerVolumeSpecName "kube-api-access-rvfqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:10.650532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.650454 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c143d7-3e4a-4b79-8080-58ed1909aab6-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:10.650532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:10.650485 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfqj\" (UniqueName: \"kubernetes.io/projected/60c143d7-3e4a-4b79-8080-58ed1909aab6-kube-api-access-rvfqj\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:11.376613 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:11.376587 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" Oct 31 01:19:11.376788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:11.376623 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4" event={"ID":"60c143d7-3e4a-4b79-8080-58ed1909aab6","Type":"ContainerDied","Data":"a5fa29dd54a46e22f1b7a10b1522eb957a87c4e1e4fb8a92b6e3a1c14d52ac9a"} Oct 31 01:19:11.376788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:11.376689 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5fa29dd54a46e22f1b7a10b1522eb957a87c4e1e4fb8a92b6e3a1c14d52ac9a" Oct 31 01:19:12.606635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:12.606600 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:19:17.045999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:17.045956 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp"] Oct 31 01:19:17.095580 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:17.095542 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af7a6df-7xlbp"] Oct 31 01:19:18.311359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:18.311327 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f53552-1f41-4aab-8543-591469dd9134" path="/var/lib/kubelet/pods/c8f53552-1f41-4aab-8543-591469dd9134/volumes" Oct 31 01:19:19.708547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.708515 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q"] Oct 31 01:19:19.709470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.709439 5756 cpu_manager.go:414] "containerMap: removing container" podUID="60c143d7-3e4a-4b79-8080-58ed1909aab6" containerName="revoke-keys" Oct 31 01:19:19.709653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.709630 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c143d7-3e4a-4b79-8080-58ed1909aab6" containerName="revoke-keys" Oct 31 01:19:19.709811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.709800 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c143d7-3e4a-4b79-8080-58ed1909aab6" containerName="revoke-keys" Oct 31 01:19:19.714737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.714711 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.769921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.769890 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q"] Oct 31 01:19:19.799327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.799293 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.799509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.799490 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgxc\" (UniqueName: \"kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.799588 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.799569 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.815938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.815909 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:19.816083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.815956 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:19.901002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.900976 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgxc\" (UniqueName: \"kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.901181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.901050 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.901181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.901128 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.901733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.901709 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.903683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.903659 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:19.957080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:19.957050 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgxc\" (UniqueName: \"kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc\") pod \"maas-api-revoke-sub-e4fa394b-s5z4q\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:20.029984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:20.029914 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:20.402515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:20.402491 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q"] Oct 31 01:19:20.404234 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:20.404200 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93516cb_00d8_45b0_bc3c_52808de0c24d.slice/crio-bdf910f6241d36f03b63618688b3ffe0907483794e0a4e9e89b75870c8caa6f3 WatchSource:0}: Error finding container bdf910f6241d36f03b63618688b3ffe0907483794e0a4e9e89b75870c8caa6f3: Status 404 returned error can't find the container with id bdf910f6241d36f03b63618688b3ffe0907483794e0a4e9e89b75870c8caa6f3 Oct 31 01:19:20.447182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:20.447145 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" event={"ID":"f93516cb-00d8-45b0-bc3c-52808de0c24d","Type":"ContainerStarted","Data":"bdf910f6241d36f03b63618688b3ffe0907483794e0a4e9e89b75870c8caa6f3"} Oct 31 01:19:21.455601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:21.455566 5756 generic.go:356] "Generic (PLEG): container finished" podID="f93516cb-00d8-45b0-bc3c-52808de0c24d" containerID="b3a4ae29d03f7fb7d28802c7a34facf852defc852b5a9de442c8ab2088e58e9e" exitCode=0 Oct 31 01:19:21.456031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:21.455647 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" event={"ID":"f93516cb-00d8-45b0-bc3c-52808de0c24d","Type":"ContainerDied","Data":"b3a4ae29d03f7fb7d28802c7a34facf852defc852b5a9de442c8ab2088e58e9e"} Oct 31 01:19:22.619712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.619687 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:22.735507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.735423 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca\") pod \"f93516cb-00d8-45b0-bc3c-52808de0c24d\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " Oct 31 01:19:22.735753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.735612 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca\") pod \"f93516cb-00d8-45b0-bc3c-52808de0c24d\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " Oct 31 01:19:22.735753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.735680 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc\" (UniqueName: \"kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc\") pod \"f93516cb-00d8-45b0-bc3c-52808de0c24d\" (UID: \"f93516cb-00d8-45b0-bc3c-52808de0c24d\") " Oct 31 01:19:22.736154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.736124 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca" pod "f93516cb-00d8-45b0-bc3c-52808de0c24d" (UID: "f93516cb-00d8-45b0-bc3c-52808de0c24d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:22.740727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.740683 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca" pod "f93516cb-00d8-45b0-bc3c-52808de0c24d" (UID: "f93516cb-00d8-45b0-bc3c-52808de0c24d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:22.740985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.740966 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc" pod "f93516cb-00d8-45b0-bc3c-52808de0c24d" (UID: "f93516cb-00d8-45b0-bc3c-52808de0c24d"). InnerVolumeSpecName "kube-api-access-swgxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:22.837131 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.837095 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:22.837131 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.837130 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swgxc\" (UniqueName: \"kubernetes.io/projected/f93516cb-00d8-45b0-bc3c-52808de0c24d-kube-api-access-swgxc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:22.837329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:22.837145 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93516cb-00d8-45b0-bc3c-52808de0c24d-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:23.469477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.469449 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" Oct 31 01:19:23.469663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.469507 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q" event={"ID":"f93516cb-00d8-45b0-bc3c-52808de0c24d","Type":"ContainerDied","Data":"bdf910f6241d36f03b63618688b3ffe0907483794e0a4e9e89b75870c8caa6f3"} Oct 31 01:19:23.469663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.469573 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf910f6241d36f03b63618688b3ffe0907483794e0a4e9e89b75870c8caa6f3" Oct 31 01:19:23.581303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.581271 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b"] Oct 31 01:19:23.581766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.581753 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f93516cb-00d8-45b0-bc3c-52808de0c24d" containerName="revoke-keys" Oct 31 01:19:23.581766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.581766 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93516cb-00d8-45b0-bc3c-52808de0c24d" containerName="revoke-keys" Oct 31 01:19:23.581940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.581911 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93516cb-00d8-45b0-bc3c-52808de0c24d" containerName="revoke-keys" Oct 31 01:19:23.585987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.585959 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.608982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.608951 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b"] Oct 31 01:19:23.640273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.640246 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:23.640769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.640646 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:23.747349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.747264 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.747349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.747310 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hbxj\" (UniqueName: \"kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.747554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.747429 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.849043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.849009 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.849043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.849051 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hbxj\" (UniqueName: \"kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.849251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.849091 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.849698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.849662 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.851668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.851642 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.906477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.906450 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hbxj\" (UniqueName: \"kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj\") pod \"maas-api-revoke-sub-92aabbf6-7vq5b\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:23.910399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:23.910375 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:24.292596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:24.292561 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b"] Oct 31 01:19:24.293329 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:24.293301 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda98d07_4f51_47a0_b44a_fa95ed4fc338.slice/crio-375aba983b2b87063493962c62af68e3a958319cfd87466dd280e815b879c562 WatchSource:0}: Error finding container 375aba983b2b87063493962c62af68e3a958319cfd87466dd280e815b879c562: Status 404 returned error can't find the container with id 375aba983b2b87063493962c62af68e3a958319cfd87466dd280e815b879c562 Oct 31 01:19:24.476717 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:24.476676 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" event={"ID":"eda98d07-4f51-47a0-b44a-fa95ed4fc338","Type":"ContainerStarted","Data":"375aba983b2b87063493962c62af68e3a958319cfd87466dd280e815b879c562"} Oct 31 01:19:25.484543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:25.484460 5756 generic.go:356] "Generic (PLEG): container finished" podID="eda98d07-4f51-47a0-b44a-fa95ed4fc338" containerID="3f3366be1094a428ba3d410cd477cfcc5cff3b4468d8c3325db553b34cd29238" exitCode=0 Oct 31 01:19:25.484925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:25.484550 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" event={"ID":"eda98d07-4f51-47a0-b44a-fa95ed4fc338","Type":"ContainerDied","Data":"3f3366be1094a428ba3d410cd477cfcc5cff3b4468d8c3325db553b34cd29238"} Oct 31 01:19:26.649184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.649158 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:26.785523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.785497 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj\" (UniqueName: \"kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj\") pod \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " Oct 31 01:19:26.785697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.785682 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca\") pod \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " Oct 31 01:19:26.785764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.785709 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca\") pod \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\" (UID: \"eda98d07-4f51-47a0-b44a-fa95ed4fc338\") " Oct 31 01:19:26.786209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.786152 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca" pod "eda98d07-4f51-47a0-b44a-fa95ed4fc338" (UID: "eda98d07-4f51-47a0-b44a-fa95ed4fc338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:26.787748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.787723 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj" pod "eda98d07-4f51-47a0-b44a-fa95ed4fc338" (UID: "eda98d07-4f51-47a0-b44a-fa95ed4fc338"). InnerVolumeSpecName "kube-api-access-7hbxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:26.787889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.787849 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca" pod "eda98d07-4f51-47a0-b44a-fa95ed4fc338" (UID: "eda98d07-4f51-47a0-b44a-fa95ed4fc338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:26.887133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.887104 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hbxj\" (UniqueName: \"kubernetes.io/projected/eda98d07-4f51-47a0-b44a-fa95ed4fc338-kube-api-access-7hbxj\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:26.887133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.887132 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:26.887327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:26.887142 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eda98d07-4f51-47a0-b44a-fa95ed4fc338-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:27.046683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:27.046593 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb"] Oct 31 01:19:27.049125 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:27.049085 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-484d0504-fzdsb"] Oct 31 01:19:27.497591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:27.497518 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" Oct 31 01:19:27.497724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:27.497511 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b" event={"ID":"eda98d07-4f51-47a0-b44a-fa95ed4fc338","Type":"ContainerDied","Data":"375aba983b2b87063493962c62af68e3a958319cfd87466dd280e815b879c562"} Oct 31 01:19:27.497724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:27.497645 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="375aba983b2b87063493962c62af68e3a958319cfd87466dd280e815b879c562" Oct 31 01:19:28.311880 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:28.311829 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e0545a-7218-4e90-8815-edd6ee0be6d7" path="/var/lib/kubelet/pods/88e0545a-7218-4e90-8815-edd6ee0be6d7/volumes" Oct 31 01:19:29.361904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.361857 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5"] Oct 31 01:19:29.362415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.362402 5756 cpu_manager.go:414] "containerMap: removing container" podUID="eda98d07-4f51-47a0-b44a-fa95ed4fc338" containerName="revoke-keys" Oct 31 01:19:29.362461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.362416 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eda98d07-4f51-47a0-b44a-fa95ed4fc338" containerName="revoke-keys" Oct 31 01:19:29.362507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.362498 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="eda98d07-4f51-47a0-b44a-fa95ed4fc338" containerName="revoke-keys" Oct 31 01:19:29.366018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.366001 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.422973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.422941 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5"] Oct 31 01:19:29.469185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.469154 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:29.469345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.469198 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:29.520117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.520084 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.520280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.520128 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xjg\" (UniqueName: \"kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.520356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.520340 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.621943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.621843 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.622088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.621952 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.622088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.621982 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xjg\" (UniqueName: \"kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.622736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.622680 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.624464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.624441 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.678702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.678670 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xjg\" (UniqueName: \"kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg\") pod \"maas-api-revoke-sub-ff67da5f-mh8r5\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:29.679441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:29.679426 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:30.056938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:30.056910 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5"] Oct 31 01:19:30.058638 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:30.058612 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0d1420_7ad4_45e7_a174_71db7c246361.slice/crio-e1c27a5b7a7f9da0282b48686ff4bd5e17dec59dfae2285d527da51b779daab4 WatchSource:0}: Error finding container e1c27a5b7a7f9da0282b48686ff4bd5e17dec59dfae2285d527da51b779daab4: Status 404 returned error can't find the container with id e1c27a5b7a7f9da0282b48686ff4bd5e17dec59dfae2285d527da51b779daab4 Oct 31 01:19:30.517561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:30.517523 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" event={"ID":"6d0d1420-7ad4-45e7-a174-71db7c246361","Type":"ContainerStarted","Data":"37fcfda1f6445646641ed5f73a7c116d9ce488f43efa5ca6fb7215f2aae5f26d"} Oct 31 01:19:30.518070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:30.517592 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" event={"ID":"6d0d1420-7ad4-45e7-a174-71db7c246361","Type":"ContainerStarted","Data":"e1c27a5b7a7f9da0282b48686ff4bd5e17dec59dfae2285d527da51b779daab4"} Oct 31 01:19:30.637250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:30.637154 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" podStartSLOduration=1.6371413320000001 podStartE2EDuration="1.637141332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:30.633791331 +0000 UTC m=+1342.440758873" watchObservedRunningTime="2026-10-31 01:19:30.637141332 +0000 UTC m=+1342.444108874" Oct 31 01:19:31.525800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:31.525769 5756 generic.go:356] "Generic (PLEG): container finished" podID="6d0d1420-7ad4-45e7-a174-71db7c246361" containerID="37fcfda1f6445646641ed5f73a7c116d9ce488f43efa5ca6fb7215f2aae5f26d" exitCode=0 Oct 31 01:19:31.526217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:31.525823 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" event={"ID":"6d0d1420-7ad4-45e7-a174-71db7c246361","Type":"ContainerDied","Data":"37fcfda1f6445646641ed5f73a7c116d9ce488f43efa5ca6fb7215f2aae5f26d"} Oct 31 01:19:32.067724 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:19:32.067687 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:19:32.689351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.689328 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:32.759140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.759109 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg\" (UniqueName: \"kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg\") pod \"6d0d1420-7ad4-45e7-a174-71db7c246361\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " Oct 31 01:19:32.759307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.759163 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca\") pod \"6d0d1420-7ad4-45e7-a174-71db7c246361\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " Oct 31 01:19:32.759307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.759271 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca\") pod \"6d0d1420-7ad4-45e7-a174-71db7c246361\" (UID: \"6d0d1420-7ad4-45e7-a174-71db7c246361\") " Oct 31 01:19:32.759705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.759671 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca" pod "6d0d1420-7ad4-45e7-a174-71db7c246361" (UID: "6d0d1420-7ad4-45e7-a174-71db7c246361"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:32.759938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.759842 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:32.761490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.761466 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca" pod "6d0d1420-7ad4-45e7-a174-71db7c246361" (UID: "6d0d1420-7ad4-45e7-a174-71db7c246361"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:32.761490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.761472 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg" pod "6d0d1420-7ad4-45e7-a174-71db7c246361" (UID: "6d0d1420-7ad4-45e7-a174-71db7c246361"). InnerVolumeSpecName "kube-api-access-h6xjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:32.860602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.860520 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xjg\" (UniqueName: \"kubernetes.io/projected/6d0d1420-7ad4-45e7-a174-71db7c246361-kube-api-access-h6xjg\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:32.860602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:32.860550 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d0d1420-7ad4-45e7-a174-71db7c246361-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:33.051059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:33.051025 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9"] Oct 31 01:19:33.054221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:33.054191 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5125b284-82rn9"] Oct 31 01:19:33.540826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:33.540785 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" event={"ID":"6d0d1420-7ad4-45e7-a174-71db7c246361","Type":"ContainerDied","Data":"e1c27a5b7a7f9da0282b48686ff4bd5e17dec59dfae2285d527da51b779daab4"} Oct 31 01:19:33.541161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:33.541138 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1c27a5b7a7f9da0282b48686ff4bd5e17dec59dfae2285d527da51b779daab4" Oct 31 01:19:33.541345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:33.541330 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5" Oct 31 01:19:34.344888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:34.338835 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9543ac-e0c4-45f2-82e5-9b1d9e276483" path="/var/lib/kubelet/pods/1a9543ac-e0c4-45f2-82e5-9b1d9e276483/volumes" Oct 31 01:19:37.716779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.716744 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc"] Oct 31 01:19:37.717304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.717289 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6d0d1420-7ad4-45e7-a174-71db7c246361" containerName="revoke-keys" Oct 31 01:19:37.717360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.717305 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0d1420-7ad4-45e7-a174-71db7c246361" containerName="revoke-keys" Oct 31 01:19:37.717407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.717399 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0d1420-7ad4-45e7-a174-71db7c246361" containerName="revoke-keys" Oct 31 01:19:37.721924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.721906 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.769495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.769470 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns657\" (UniqueName: \"kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.769641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.769515 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.769689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.769648 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.774082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.774053 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:37.823282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.823254 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:37.825872 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.825844 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc"] Oct 31 01:19:37.871068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.871041 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns657\" (UniqueName: \"kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.871193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.871085 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.871193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.871174 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.871850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.871818 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.873517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.873497 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:37.927333 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:37.927288 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns657\" (UniqueName: \"kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657\") pod \"maas-api-revoke-sub-81f72634-klrtc\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:38.035116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:38.035087 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:38.407976 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:38.407946 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc"] Oct 31 01:19:38.409223 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:38.409194 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cef7c3e_0993_4d3d_9ead_be79531443ce.slice/crio-a44f4cb4271652d17d4a85a1ae4f28d54678884216e029ae081db6ba17d139e2 WatchSource:0}: Error finding container a44f4cb4271652d17d4a85a1ae4f28d54678884216e029ae081db6ba17d139e2: Status 404 returned error can't find the container with id a44f4cb4271652d17d4a85a1ae4f28d54678884216e029ae081db6ba17d139e2 Oct 31 01:19:38.588025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:38.587976 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" event={"ID":"2cef7c3e-0993-4d3d-9ead-be79531443ce","Type":"ContainerStarted","Data":"a44f4cb4271652d17d4a85a1ae4f28d54678884216e029ae081db6ba17d139e2"} Oct 31 01:19:39.595765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:39.595678 5756 generic.go:356] "Generic (PLEG): container finished" podID="2cef7c3e-0993-4d3d-9ead-be79531443ce" containerID="fd7b4377ab9dca58f97dd6a56a19277e8caa4522995d07991f4f3e469263f6ba" exitCode=0 Oct 31 01:19:39.595765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:39.595744 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" event={"ID":"2cef7c3e-0993-4d3d-9ead-be79531443ce","Type":"ContainerDied","Data":"fd7b4377ab9dca58f97dd6a56a19277e8caa4522995d07991f4f3e469263f6ba"} Oct 31 01:19:40.761810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.761788 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:40.807496 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.807468 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca\") pod \"2cef7c3e-0993-4d3d-9ead-be79531443ce\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " Oct 31 01:19:40.807627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.807520 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca\") pod \"2cef7c3e-0993-4d3d-9ead-be79531443ce\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " Oct 31 01:19:40.807815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.807796 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657\" (UniqueName: \"kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657\") pod \"2cef7c3e-0993-4d3d-9ead-be79531443ce\" (UID: \"2cef7c3e-0993-4d3d-9ead-be79531443ce\") " Oct 31 01:19:40.807909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.807883 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca" pod "2cef7c3e-0993-4d3d-9ead-be79531443ce" (UID: "2cef7c3e-0993-4d3d-9ead-be79531443ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:40.808278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.808210 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:40.809708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.809684 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca" pod "2cef7c3e-0993-4d3d-9ead-be79531443ce" (UID: "2cef7c3e-0993-4d3d-9ead-be79531443ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:40.809844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.809830 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657" pod "2cef7c3e-0993-4d3d-9ead-be79531443ce" (UID: "2cef7c3e-0993-4d3d-9ead-be79531443ce"). InnerVolumeSpecName "kube-api-access-ns657". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:40.909493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.909404 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns657\" (UniqueName: \"kubernetes.io/projected/2cef7c3e-0993-4d3d-9ead-be79531443ce-kube-api-access-ns657\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:40.909493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:40.909431 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cef7c3e-0993-4d3d-9ead-be79531443ce-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:41.609166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:41.609134 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" Oct 31 01:19:41.609352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:41.609161 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc" event={"ID":"2cef7c3e-0993-4d3d-9ead-be79531443ce","Type":"ContainerDied","Data":"a44f4cb4271652d17d4a85a1ae4f28d54678884216e029ae081db6ba17d139e2"} Oct 31 01:19:41.609352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:41.609216 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44f4cb4271652d17d4a85a1ae4f28d54678884216e029ae081db6ba17d139e2" Oct 31 01:19:42.050572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.050521 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf"] Oct 31 01:19:42.053209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.053186 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d7f4c3-mzfqf"] Oct 31 01:19:42.311889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.311796 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d52f99-7f8d-4965-b710-feb93f51668f" path="/var/lib/kubelet/pods/98d52f99-7f8d-4965-b710-feb93f51668f/volumes" Oct 31 01:19:42.783509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.783480 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc"] Oct 31 01:19:42.784037 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.784023 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2cef7c3e-0993-4d3d-9ead-be79531443ce" containerName="revoke-keys" Oct 31 01:19:42.784099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.784039 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cef7c3e-0993-4d3d-9ead-be79531443ce" containerName="revoke-keys" Oct 31 01:19:42.784145 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.784136 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cef7c3e-0993-4d3d-9ead-be79531443ce" containerName="revoke-keys" Oct 31 01:19:42.787516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.787499 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833019 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833057 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfpb\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833180 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833319 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833405 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833490 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833537 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833565 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.833634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.833608 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.848776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.848742 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc"] Oct 31 01:19:42.935338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935298 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935349 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935402 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935426 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfpb\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935473 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935576 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935683 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935720 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935743 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935794 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.935907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.935813 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.936154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.936048 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.936271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.936249 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.936326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.936306 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.938189 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.938168 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.938620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.938601 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.992329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.992303 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:42.992500 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:42.992479 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfpb\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:43.101520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.101432 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:43.208930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.208891 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql"] Oct 31 01:19:43.214762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.214737 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.267452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.267414 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:43.267667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.267527 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:43.271328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.271298 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql"] Oct 31 01:19:43.340558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.340526 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db9k5\" (UniqueName: \"kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.340726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.340662 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.340766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.340729 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.441751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.441675 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.441751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.441740 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.441959 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.441817 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db9k5\" (UniqueName: \"kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.442589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.442524 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.444168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.444149 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.499682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.499649 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db9k5\" (UniqueName: \"kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5\") pod \"maas-api-revoke-sub-2623bb2e-bdqql\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.530538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.530509 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:43.532468 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.532440 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc"] Oct 31 01:19:43.534140 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:43.534111 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e0aac8_3fbf_4463_b067_22a0c2943388.slice/crio-a0c4aa21f2d923789b029fdaa3d7b30a06112d88201649fd116a80e55e23e759 WatchSource:0}: Error finding container a0c4aa21f2d923789b029fdaa3d7b30a06112d88201649fd116a80e55e23e759: Status 404 returned error can't find the container with id a0c4aa21f2d923789b029fdaa3d7b30a06112d88201649fd116a80e55e23e759 Oct 31 01:19:43.540476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.540446 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:43.540588 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.540510 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:43.540588 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.540564 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:43.630447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.630388 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" event={"ID":"d3e0aac8-3fbf-4463-b067-22a0c2943388","Type":"ContainerStarted","Data":"a0c4aa21f2d923789b029fdaa3d7b30a06112d88201649fd116a80e55e23e759"} Oct 31 01:19:43.933065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:43.933035 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql"] Oct 31 01:19:43.933196 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:43.933150 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153ffb2f_d944_4be3_9a2d_6105864ac62b.slice/crio-c7d92cf0a9575e97a2dfd875ed66c217753f9a0909b5af87afd1d996d0b0f43e WatchSource:0}: Error finding container c7d92cf0a9575e97a2dfd875ed66c217753f9a0909b5af87afd1d996d0b0f43e: Status 404 returned error can't find the container with id c7d92cf0a9575e97a2dfd875ed66c217753f9a0909b5af87afd1d996d0b0f43e Oct 31 01:19:44.659357 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:44.659257 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" event={"ID":"d3e0aac8-3fbf-4463-b067-22a0c2943388","Type":"ContainerStarted","Data":"c7429407961e146713ca2abfacc09284aff9920c72899abf5889de776b440b1f"} Oct 31 01:19:44.660943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:44.660919 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" event={"ID":"153ffb2f-d944-4be3-9a2d-6105864ac62b","Type":"ContainerStarted","Data":"f8c41d47989db0608ac8301ff91c53d0a8f7fac662ba3e7632d879f4fd6d6ef2"} Oct 31 01:19:44.661056 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:44.660959 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" event={"ID":"153ffb2f-d944-4be3-9a2d-6105864ac62b","Type":"ContainerStarted","Data":"c7d92cf0a9575e97a2dfd875ed66c217753f9a0909b5af87afd1d996d0b0f43e"} Oct 31 01:19:44.783256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:44.783194 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podStartSLOduration=2.783175606 podStartE2EDuration="2.783175606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:44.780788361 +0000 UTC m=+1356.587755906" watchObservedRunningTime="2026-10-31 01:19:44.783175606 +0000 UTC m=+1356.590143150" Oct 31 01:19:44.899880 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:44.899812 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" podStartSLOduration=1.8997980650000001 podStartE2EDuration="1.899798065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:44.897675742 +0000 UTC m=+1356.704643335" watchObservedRunningTime="2026-10-31 01:19:44.899798065 +0000 UTC m=+1356.706765607" Oct 31 01:19:45.102298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:45.102268 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:45.104202 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:45.104172 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:19:45.104358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:45.104228 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 01:19:45.668393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:45.668316 5756 generic.go:356] "Generic (PLEG): container finished" podID="153ffb2f-d944-4be3-9a2d-6105864ac62b" containerID="f8c41d47989db0608ac8301ff91c53d0a8f7fac662ba3e7632d879f4fd6d6ef2" exitCode=0 Oct 31 01:19:45.668737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:45.668394 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" event={"ID":"153ffb2f-d944-4be3-9a2d-6105864ac62b","Type":"ContainerDied","Data":"f8c41d47989db0608ac8301ff91c53d0a8f7fac662ba3e7632d879f4fd6d6ef2"} Oct 31 01:19:46.102489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.102450 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:19:46.102642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.102520 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 01:19:46.862378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.862348 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:46.887067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.887042 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca\") pod \"153ffb2f-d944-4be3-9a2d-6105864ac62b\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " Oct 31 01:19:46.887179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.887125 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca\") pod \"153ffb2f-d944-4be3-9a2d-6105864ac62b\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " Oct 31 01:19:46.887277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.887260 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5\" (UniqueName: \"kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5\") pod \"153ffb2f-d944-4be3-9a2d-6105864ac62b\" (UID: \"153ffb2f-d944-4be3-9a2d-6105864ac62b\") " Oct 31 01:19:46.887447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.887424 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca" pod "153ffb2f-d944-4be3-9a2d-6105864ac62b" (UID: "153ffb2f-d944-4be3-9a2d-6105864ac62b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:46.887694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.887677 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:46.889343 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.889323 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5" pod "153ffb2f-d944-4be3-9a2d-6105864ac62b" (UID: "153ffb2f-d944-4be3-9a2d-6105864ac62b"). InnerVolumeSpecName "kube-api-access-db9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:46.889426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.889408 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca" pod "153ffb2f-d944-4be3-9a2d-6105864ac62b" (UID: "153ffb2f-d944-4be3-9a2d-6105864ac62b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:46.988642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.988562 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db9k5\" (UniqueName: \"kubernetes.io/projected/153ffb2f-d944-4be3-9a2d-6105864ac62b-kube-api-access-db9k5\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:46.988642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:46.988593 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/153ffb2f-d944-4be3-9a2d-6105864ac62b-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:47.101731 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:47.101698 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:19:47.101956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:47.101752 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 01:19:47.683486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:47.683444 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" event={"ID":"153ffb2f-d944-4be3-9a2d-6105864ac62b","Type":"ContainerDied","Data":"c7d92cf0a9575e97a2dfd875ed66c217753f9a0909b5af87afd1d996d0b0f43e"} Oct 31 01:19:47.683789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:47.683478 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql" Oct 31 01:19:47.683789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:47.683526 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7d92cf0a9575e97a2dfd875ed66c217753f9a0909b5af87afd1d996d0b0f43e" Oct 31 01:19:48.101957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.101921 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:19:48.102421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.101964 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 01:19:48.512027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.511943 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj"] Oct 31 01:19:48.512513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.512498 5756 cpu_manager.go:414] "containerMap: removing container" podUID="153ffb2f-d944-4be3-9a2d-6105864ac62b" containerName="revoke-keys" Oct 31 01:19:48.512561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.512515 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153ffb2f-d944-4be3-9a2d-6105864ac62b" containerName="revoke-keys" Oct 31 01:19:48.512606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.512597 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="153ffb2f-d944-4be3-9a2d-6105864ac62b" containerName="revoke-keys" Oct 31 01:19:48.515898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.515876 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.568509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.568481 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:19:48.573206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.573180 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj"] Oct 31 01:19:48.610211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.610177 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.610392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.610321 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkv6\" (UniqueName: \"kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.610471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.610411 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.617353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.617331 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:19:48.712091 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.712040 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.712257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.712159 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkv6\" (UniqueName: \"kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.712257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.712244 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.713103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.713074 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.715079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.715049 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.773210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.773144 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkv6\" (UniqueName: \"kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6\") pod \"maas-api-revoke-sub-41f35019-8s4sj\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:48.852226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:48.852198 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:49.102779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:49.102749 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:19:49.103177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:49.102802 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 01:19:49.255967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:49.255852 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj"] Oct 31 01:19:49.256360 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:49.256332 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e593ea0_02ea_402c_ba1b_2249cc3c1a2f.slice/crio-51de93dcfacc673d9b737bf2c107e27d7e38c82c762b0df0f550bee814cc652c WatchSource:0}: Error finding container 51de93dcfacc673d9b737bf2c107e27d7e38c82c762b0df0f550bee814cc652c: Status 404 returned error can't find the container with id 51de93dcfacc673d9b737bf2c107e27d7e38c82c762b0df0f550bee814cc652c Oct 31 01:19:49.697768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:49.697730 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" event={"ID":"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f","Type":"ContainerStarted","Data":"51de93dcfacc673d9b737bf2c107e27d7e38c82c762b0df0f550bee814cc652c"} Oct 31 01:19:50.101903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:50.101843 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:19:50.102087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:50.101923 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 01:19:50.706121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:50.706081 5756 generic.go:356] "Generic (PLEG): container finished" podID="5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" containerID="be1a0568d274dbc14322580496a2e5d660158c774e9f162991e39123d21f3bad" exitCode=0 Oct 31 01:19:50.706655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:50.706167 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" event={"ID":"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f","Type":"ContainerDied","Data":"be1a0568d274dbc14322580496a2e5d660158c774e9f162991e39123d21f3bad"} Oct 31 01:19:51.105909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.105878 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:51.106298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.106278 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:51.107002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.106982 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:19:51.878045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.878021 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:51.956653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.956623 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca\") pod \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " Oct 31 01:19:51.956824 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.956805 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca\") pod \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " Oct 31 01:19:51.956930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.956914 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6\" (UniqueName: \"kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6\") pod \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\" (UID: \"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f\") " Oct 31 01:19:51.957019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.956993 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca" pod "5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" (UID: "5e593ea0-02ea-402c-ba1b-2249cc3c1a2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:19:51.957432 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.957398 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:51.959019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.958993 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca" pod "5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" (UID: "5e593ea0-02ea-402c-ba1b-2249cc3c1a2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:19:51.959121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:51.959071 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6" pod "5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" (UID: "5e593ea0-02ea-402c-ba1b-2249cc3c1a2f"). InnerVolumeSpecName "kube-api-access-jjkv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:19:52.058182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:52.058140 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:52.058182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:52.058176 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjkv6\" (UniqueName: \"kubernetes.io/projected/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f-kube-api-access-jjkv6\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:19:52.720328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:52.720244 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" Oct 31 01:19:52.720328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:52.720277 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj" event={"ID":"5e593ea0-02ea-402c-ba1b-2249cc3c1a2f","Type":"ContainerDied","Data":"51de93dcfacc673d9b737bf2c107e27d7e38c82c762b0df0f550bee814cc652c"} Oct 31 01:19:52.720328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:52.720322 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51de93dcfacc673d9b737bf2c107e27d7e38c82c762b0df0f550bee814cc652c" Oct 31 01:19:53.050504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:53.050471 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff"] Oct 31 01:19:53.063780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:53.063752 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32dfa249-4vzff"] Oct 31 01:19:54.311383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:54.311356 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd3b862-ba18-4a4b-94d7-08a2a1972a67" path="/var/lib/kubelet/pods/1cd3b862-ba18-4a4b-94d7-08a2a1972a67/volumes" Oct 31 01:19:55.849928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:55.849893 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:19:56.605975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.605932 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn"] Oct 31 01:19:56.607295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.607255 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" containerName="revoke-keys" Oct 31 01:19:56.607476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.607459 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" containerName="revoke-keys" Oct 31 01:19:56.607773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.607757 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" containerName="revoke-keys" Oct 31 01:19:56.613639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.613621 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.723685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.721099 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn"] Oct 31 01:19:56.741965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.736182 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.741965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.736501 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx76t\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.741965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.736696 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.741965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.736779 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.751953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.736934 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.751953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.750041 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.751953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.750109 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.751953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.750180 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.751953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.750241 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.896003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.895902 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx76t\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.899043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.899004 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.912055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.912016 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.933832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.933626 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.934146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.934068 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.934315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.934298 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.941074 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.941031 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.954010 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.941563 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.954567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.954206 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.954801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.954782 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.976240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.954501 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.976240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.963352 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:56.976513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:56.976488 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:57.050890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:57.026533 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:57.050890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:57.041211 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:57.063889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:57.061317 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx76t\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:57.084887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:57.072415 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:57.139701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:57.139437 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:57.279270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:57.279171 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:19:58.533561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.501046 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:19:58.627898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.617554 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 01:19:58.648886 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:19:58.646918 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42112337_beed_46a8_b4d6_3f69f46c68f1.slice/crio-fc3ce3974292e2b8a71b4faeb90568cb65a4ae5cbe3bd9cb9a2e4727e2e15b6f WatchSource:0}: Error finding container fc3ce3974292e2b8a71b4faeb90568cb65a4ae5cbe3bd9cb9a2e4727e2e15b6f: Status 404 returned error can't find the container with id fc3ce3974292e2b8a71b4faeb90568cb65a4ae5cbe3bd9cb9a2e4727e2e15b6f Oct 31 01:19:58.648886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.647306 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn"] Oct 31 01:19:58.677901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.676707 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:58.677901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.676805 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:58.677901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.676848 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:19:58.743514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.742630 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:19:58.753991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.753948 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 01:19:58.802609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.802256 5756 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 31 01:19:58.911777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.911727 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:19:58.972899 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:58.971361 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" event={"ID":"42112337-beed-46a8-b4d6-3f69f46c68f1","Type":"ContainerStarted","Data":"fc3ce3974292e2b8a71b4faeb90568cb65a4ae5cbe3bd9cb9a2e4727e2e15b6f"} Oct 31 01:19:59.501272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:59.501172 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:19:59.998915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:19:59.995112 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" event={"ID":"42112337-beed-46a8-b4d6-3f69f46c68f1","Type":"ContainerStarted","Data":"a18c48dc251f214495636c773539f193aedaf3b01c183d8701111cffa90ef374"} Oct 31 01:20:00.166699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:00.166624 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podStartSLOduration=4.166602893 podStartE2EDuration="4.166602893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:00.163631831 +0000 UTC m=+1371.970599376" watchObservedRunningTime="2026-10-31 01:20:00.166602893 +0000 UTC m=+1371.973570437" Oct 31 01:20:00.280341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:00.280257 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:20:00.282162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:00.282130 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:00.282401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:00.282370 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:01.280041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:01.280008 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:01.280428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:01.280064 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:02.080599 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:20:02.080562 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:20:02.280342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:02.280307 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:02.280755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:02.280382 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:03.280455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:03.280423 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:03.281058 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:03.280476 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:04.280288 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:04.280253 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:04.280504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:04.280308 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:05.280244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:05.280209 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:05.280449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:05.280273 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:06.280271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.280240 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:20:06.280658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.280304 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 01:20:06.317117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.317086 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s"] Oct 31 01:20:06.321555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.321532 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.374600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.374572 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:20:06.378376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.378351 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:06.452293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.452255 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.452684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.452658 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcf5l\" (UniqueName: \"kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.454927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.452839 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.457829 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.457799 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s"] Oct 31 01:20:06.554548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.554518 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcf5l\" (UniqueName: \"kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.554773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.554752 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.555021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.555000 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.555813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.555780 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.557650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.557586 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.612675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.612646 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcf5l\" (UniqueName: \"kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l\") pod \"maas-api-revoke-sub-98a3be0e-7gn8s\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:06.635194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:06.635166 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:07.019470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:07.019447 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s"] Oct 31 01:20:07.022207 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:07.022174 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18a41c7_983a_49b2_81d1_699f9fbcc4f3.slice/crio-2706d341959c05728e36365926cdc05dc64e4dbdc43ba6995e895fd4e5f278ad WatchSource:0}: Error finding container 2706d341959c05728e36365926cdc05dc64e4dbdc43ba6995e895fd4e5f278ad: Status 404 returned error can't find the container with id 2706d341959c05728e36365926cdc05dc64e4dbdc43ba6995e895fd4e5f278ad Oct 31 01:20:07.047561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:07.047390 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" event={"ID":"c18a41c7-983a-49b2-81d1-699f9fbcc4f3","Type":"ContainerStarted","Data":"2706d341959c05728e36365926cdc05dc64e4dbdc43ba6995e895fd4e5f278ad"} Oct 31 01:20:07.363231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:07.363199 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:20:07.363641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:07.363603 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:20:07.364339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:07.364321 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:20:08.055665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:08.055629 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" event={"ID":"c18a41c7-983a-49b2-81d1-699f9fbcc4f3","Type":"ContainerStarted","Data":"9c9ef3baed98ae28065460237ac8706ab1559ef8ff652207a94fbb045229f517"} Oct 31 01:20:08.172503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:08.172438 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" podStartSLOduration=2.172419556 podStartE2EDuration="2.172419556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:08.168668695 +0000 UTC m=+1379.975636255" watchObservedRunningTime="2026-10-31 01:20:08.172419556 +0000 UTC m=+1379.979387104" Oct 31 01:20:08.579898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:08.579848 5756 scope.go:122] "RemoveContainer" containerID="6a4560e52603d459c7a270a3c8eecdf494d0d2fb826a8c77496674f74dd94d08" Oct 31 01:20:09.063191 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:09.063156 5756 generic.go:356] "Generic (PLEG): container finished" podID="c18a41c7-983a-49b2-81d1-699f9fbcc4f3" containerID="9c9ef3baed98ae28065460237ac8706ab1559ef8ff652207a94fbb045229f517" exitCode=0 Oct 31 01:20:09.063356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:09.063238 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" event={"ID":"c18a41c7-983a-49b2-81d1-699f9fbcc4f3","Type":"ContainerDied","Data":"9c9ef3baed98ae28065460237ac8706ab1559ef8ff652207a94fbb045229f517"} Oct 31 01:20:10.226802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.226772 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:10.304843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.304806 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca\") pod \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " Oct 31 01:20:10.305061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.304935 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l\" (UniqueName: \"kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l\") pod \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " Oct 31 01:20:10.305061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.305033 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca\") pod \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\" (UID: \"c18a41c7-983a-49b2-81d1-699f9fbcc4f3\") " Oct 31 01:20:10.305240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.305218 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca" pod "c18a41c7-983a-49b2-81d1-699f9fbcc4f3" (UID: "c18a41c7-983a-49b2-81d1-699f9fbcc4f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:10.305539 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.305517 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:10.307364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.307334 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l" pod "c18a41c7-983a-49b2-81d1-699f9fbcc4f3" (UID: "c18a41c7-983a-49b2-81d1-699f9fbcc4f3"). InnerVolumeSpecName "kube-api-access-xcf5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:10.307485 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.307436 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca" pod "c18a41c7-983a-49b2-81d1-699f9fbcc4f3" (UID: "c18a41c7-983a-49b2-81d1-699f9fbcc4f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:10.407198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.407159 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcf5l\" (UniqueName: \"kubernetes.io/projected/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-kube-api-access-xcf5l\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:10.407198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:10.407191 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18a41c7-983a-49b2-81d1-699f9fbcc4f3-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:11.077269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:11.077240 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" Oct 31 01:20:11.077429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:11.077235 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s" event={"ID":"c18a41c7-983a-49b2-81d1-699f9fbcc4f3","Type":"ContainerDied","Data":"2706d341959c05728e36365926cdc05dc64e4dbdc43ba6995e895fd4e5f278ad"} Oct 31 01:20:11.077429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:11.077368 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2706d341959c05728e36365926cdc05dc64e4dbdc43ba6995e895fd4e5f278ad" Oct 31 01:20:12.857712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.857677 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z"] Oct 31 01:20:12.858261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.858247 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c18a41c7-983a-49b2-81d1-699f9fbcc4f3" containerName="revoke-keys" Oct 31 01:20:12.858306 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.858263 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18a41c7-983a-49b2-81d1-699f9fbcc4f3" containerName="revoke-keys" Oct 31 01:20:12.858355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.858347 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18a41c7-983a-49b2-81d1-699f9fbcc4f3" containerName="revoke-keys" Oct 31 01:20:12.861889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.861874 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:12.919662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.919631 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z"] Oct 31 01:20:12.938458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.938415 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:12.938591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.938474 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ccp\" (UniqueName: \"kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:12.938591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.938529 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:12.965526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.965496 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:12.965668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:12.965536 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:20:13.040357 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.040328 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.040522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.040367 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ccp\" (UniqueName: \"kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.040522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.040400 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.041179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.041153 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.042809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.042789 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.103840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.103810 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ccp\" (UniqueName: \"kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp\") pod \"maas-api-revoke-sub-53340def-6fl5z\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.175076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.174965 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:13.575011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:13.574989 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z"] Oct 31 01:20:13.577456 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:13.577426 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8650adf2_94a1_43d1_815e_1c525cd845ca.slice/crio-ff5a69d262e71e914e784802e5cf43de90ee962ebffe76fdf58f634acff4ad92 WatchSource:0}: Error finding container ff5a69d262e71e914e784802e5cf43de90ee962ebffe76fdf58f634acff4ad92: Status 404 returned error can't find the container with id ff5a69d262e71e914e784802e5cf43de90ee962ebffe76fdf58f634acff4ad92 Oct 31 01:20:14.049291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:14.049261 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz"] Oct 31 01:20:14.100184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:14.098909 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" event={"ID":"8650adf2-94a1-43d1-815e-1c525cd845ca","Type":"ContainerStarted","Data":"acdbd4fc5e7023a9fd4c799c6fea2e8aa1d0a3c98e7f1aa484ab07b8066e26d0"} Oct 31 01:20:14.100184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:14.098974 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" event={"ID":"8650adf2-94a1-43d1-815e-1c525cd845ca","Type":"ContainerStarted","Data":"ff5a69d262e71e914e784802e5cf43de90ee962ebffe76fdf58f634acff4ad92"} Oct 31 01:20:14.100184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:14.099000 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b07904f-h9cjz"] Oct 31 01:20:14.221700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:14.221646 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" podStartSLOduration=2.221631696 podStartE2EDuration="2.221631696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:14.221482466 +0000 UTC m=+1386.028450010" watchObservedRunningTime="2026-10-31 01:20:14.221631696 +0000 UTC m=+1386.028599238" Oct 31 01:20:14.312168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:14.312135 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ebb645d-1c2b-4fd0-aad7-925c81c806c4" path="/var/lib/kubelet/pods/9ebb645d-1c2b-4fd0-aad7-925c81c806c4/volumes" Oct 31 01:20:15.108318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:15.108284 5756 generic.go:356] "Generic (PLEG): container finished" podID="8650adf2-94a1-43d1-815e-1c525cd845ca" containerID="acdbd4fc5e7023a9fd4c799c6fea2e8aa1d0a3c98e7f1aa484ab07b8066e26d0" exitCode=0 Oct 31 01:20:15.108722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:15.108377 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" event={"ID":"8650adf2-94a1-43d1-815e-1c525cd845ca","Type":"ContainerDied","Data":"acdbd4fc5e7023a9fd4c799c6fea2e8aa1d0a3c98e7f1aa484ab07b8066e26d0"} Oct 31 01:20:16.263904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.263880 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:16.387081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.386471 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp\" (UniqueName: \"kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp\") pod \"8650adf2-94a1-43d1-815e-1c525cd845ca\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " Oct 31 01:20:16.387081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.386677 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca\") pod \"8650adf2-94a1-43d1-815e-1c525cd845ca\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " Oct 31 01:20:16.387081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.386821 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca\") pod \"8650adf2-94a1-43d1-815e-1c525cd845ca\" (UID: \"8650adf2-94a1-43d1-815e-1c525cd845ca\") " Oct 31 01:20:16.388524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.388493 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca" pod "8650adf2-94a1-43d1-815e-1c525cd845ca" (UID: "8650adf2-94a1-43d1-815e-1c525cd845ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:16.390309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.390282 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp" pod "8650adf2-94a1-43d1-815e-1c525cd845ca" (UID: "8650adf2-94a1-43d1-815e-1c525cd845ca"). InnerVolumeSpecName "kube-api-access-b4ccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:16.391159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.391128 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca" pod "8650adf2-94a1-43d1-815e-1c525cd845ca" (UID: "8650adf2-94a1-43d1-815e-1c525cd845ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:16.489360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.489325 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:16.489360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.489355 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ccp\" (UniqueName: \"kubernetes.io/projected/8650adf2-94a1-43d1-815e-1c525cd845ca-kube-api-access-b4ccp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:16.489360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:16.489365 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8650adf2-94a1-43d1-815e-1c525cd845ca-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:17.089631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.089598 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4"] Oct 31 01:20:17.090317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.090292 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8650adf2-94a1-43d1-815e-1c525cd845ca" containerName="revoke-keys" Oct 31 01:20:17.090317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.090313 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8650adf2-94a1-43d1-815e-1c525cd845ca" containerName="revoke-keys" Oct 31 01:20:17.090544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.090428 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8650adf2-94a1-43d1-815e-1c525cd845ca" containerName="revoke-keys" Oct 31 01:20:17.097323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.097093 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.123934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.123911 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" Oct 31 01:20:17.123934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.123913 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z" event={"ID":"8650adf2-94a1-43d1-815e-1c525cd845ca","Type":"ContainerDied","Data":"ff5a69d262e71e914e784802e5cf43de90ee962ebffe76fdf58f634acff4ad92"} Oct 31 01:20:17.124125 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.123986 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff5a69d262e71e914e784802e5cf43de90ee962ebffe76fdf58f634acff4ad92" Oct 31 01:20:17.173520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.173491 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4"] Oct 31 01:20:17.201563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.201520 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pvqv\" (UniqueName: \"kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.201745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.201589 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.201807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.201755 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.302794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.302761 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pvqv\" (UniqueName: \"kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.303258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.302812 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.303258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.302937 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.303645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.303623 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.305321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.305303 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.361933 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.361832 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pvqv\" (UniqueName: \"kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv\") pod \"maas-api-revoke-sub-ea9a848b-hbbx4\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.414191 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.414165 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:17.800675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:17.800651 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4"] Oct 31 01:20:17.802364 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:17.802333 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec4c76a_2537_4252_879e_610d34abcff5.slice/crio-a7a1e2f4b819f32866029259135d44823c58041f23abc873dbfbb8c0526073ac WatchSource:0}: Error finding container a7a1e2f4b819f32866029259135d44823c58041f23abc873dbfbb8c0526073ac: Status 404 returned error can't find the container with id a7a1e2f4b819f32866029259135d44823c58041f23abc873dbfbb8c0526073ac Oct 31 01:20:18.131636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:18.131604 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" event={"ID":"2ec4c76a-2537-4252-879e-610d34abcff5","Type":"ContainerStarted","Data":"a7a1e2f4b819f32866029259135d44823c58041f23abc873dbfbb8c0526073ac"} Oct 31 01:20:19.141445 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:19.141410 5756 generic.go:356] "Generic (PLEG): container finished" podID="2ec4c76a-2537-4252-879e-610d34abcff5" containerID="b7839b54a5099cfe8e378b87dcb66d28d0870ccb251931c4627085bdfd0c5615" exitCode=0 Oct 31 01:20:19.143280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:19.141555 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" event={"ID":"2ec4c76a-2537-4252-879e-610d34abcff5","Type":"ContainerDied","Data":"b7839b54a5099cfe8e378b87dcb66d28d0870ccb251931c4627085bdfd0c5615"} Oct 31 01:20:20.296284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.296257 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:20.446567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.446481 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv\" (UniqueName: \"kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv\") pod \"2ec4c76a-2537-4252-879e-610d34abcff5\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " Oct 31 01:20:20.446943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.446921 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca\") pod \"2ec4c76a-2537-4252-879e-610d34abcff5\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " Oct 31 01:20:20.447101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.447086 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca\") pod \"2ec4c76a-2537-4252-879e-610d34abcff5\" (UID: \"2ec4c76a-2537-4252-879e-610d34abcff5\") " Oct 31 01:20:20.447515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.447475 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca" pod "2ec4c76a-2537-4252-879e-610d34abcff5" (UID: "2ec4c76a-2537-4252-879e-610d34abcff5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:20.447633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.447596 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:20.449077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.449047 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv" pod "2ec4c76a-2537-4252-879e-610d34abcff5" (UID: "2ec4c76a-2537-4252-879e-610d34abcff5"). InnerVolumeSpecName "kube-api-access-7pvqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:20.449705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.449682 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca" pod "2ec4c76a-2537-4252-879e-610d34abcff5" (UID: "2ec4c76a-2537-4252-879e-610d34abcff5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:20.548576 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.548546 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pvqv\" (UniqueName: \"kubernetes.io/projected/2ec4c76a-2537-4252-879e-610d34abcff5-kube-api-access-7pvqv\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:20.548576 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:20.548574 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ec4c76a-2537-4252-879e-610d34abcff5-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:21.156431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:21.156402 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" Oct 31 01:20:21.156603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:21.156440 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4" event={"ID":"2ec4c76a-2537-4252-879e-610d34abcff5","Type":"ContainerDied","Data":"a7a1e2f4b819f32866029259135d44823c58041f23abc873dbfbb8c0526073ac"} Oct 31 01:20:21.156603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:21.156486 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a1e2f4b819f32866029259135d44823c58041f23abc873dbfbb8c0526073ac" Oct 31 01:20:23.743118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:23.741954 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:23.747984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:23.747940 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:23.900022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:23.899954 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:23.933899 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:23.921950 5756 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.091028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.090977 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.282591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.282502 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.450949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.450021 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.755379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.755276 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.756472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.756449 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:20:24.757448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.755977 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 31 01:20:24.786643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.786613 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:20:24.928010 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.921642 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:24.928010 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:24.922387 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:20:25.085890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.067654 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:20:25.136121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.136058 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:25.141111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.141085 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:20:25.304522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.304478 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:25.305187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.305163 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:20:25.452294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.450907 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded" Oct 31 01:20:25.471615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.453031 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:20:25.657786 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.655180 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 01:20:25.674921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.671756 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 01:20:25.766067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.765964 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 01:20:25.799889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:25.785193 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:26.110931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.097226 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:26.139119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.139062 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 01:20:26.364946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.341575 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:26.699940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.679108 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:26.699940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.679340 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:20:26.699940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.694555 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:26.790457 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.787107 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded" Oct 31 01:20:26.830064 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.830015 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:26.848228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.846489 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 01:20:26.962035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:26.961952 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 01:20:27.019001 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:27.018891 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 01:20:27.019534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:27.019044 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 01:20:27.023700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:27.023648 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 01:20:30.158078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.158037 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln"] Oct 31 01:20:30.158778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.158756 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2ec4c76a-2537-4252-879e-610d34abcff5" containerName="revoke-keys" Oct 31 01:20:30.158892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.158780 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec4c76a-2537-4252-879e-610d34abcff5" containerName="revoke-keys" Oct 31 01:20:30.158997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.158983 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec4c76a-2537-4252-879e-610d34abcff5" containerName="revoke-keys" Oct 31 01:20:30.162372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.162345 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.215800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.215773 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:30.215995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.215815 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:20:30.268300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.268270 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln"] Oct 31 01:20:30.277958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.277926 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.278106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.277964 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.278106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.278051 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krp6m\" (UniqueName: \"kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.379300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.379269 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.379300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.379304 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.379566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.379329 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krp6m\" (UniqueName: \"kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.380033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.380009 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.382024 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.381991 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.448664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.448590 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krp6m\" (UniqueName: \"kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m\") pod \"maas-api-revoke-sub-b65a9483-45kln\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.475593 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.475562 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:30.858965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:30.858924 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln"] Oct 31 01:20:30.859083 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:30.858932 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55808545_23d5_4d0e_8d7f_f1364516c51b.slice/crio-679157006f76a94102df0b6e62c1c08f081d87488dea91293e9d85342fac7a66 WatchSource:0}: Error finding container 679157006f76a94102df0b6e62c1c08f081d87488dea91293e9d85342fac7a66: Status 404 returned error can't find the container with id 679157006f76a94102df0b6e62c1c08f081d87488dea91293e9d85342fac7a66 Oct 31 01:20:31.699218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:31.699186 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" event={"ID":"55808545-23d5-4d0e-8d7f-f1364516c51b","Type":"ContainerStarted","Data":"60a3a8f0748c570664b9615411f386be346ec09ba911e4e92f338d87864ae5c8"} Oct 31 01:20:31.699628 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:31.699235 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" event={"ID":"55808545-23d5-4d0e-8d7f-f1364516c51b","Type":"ContainerStarted","Data":"679157006f76a94102df0b6e62c1c08f081d87488dea91293e9d85342fac7a66"} Oct 31 01:20:31.886584 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:31.886533 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" podStartSLOduration=1.886518493 podStartE2EDuration="1.886518493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:31.883241469 +0000 UTC m=+1403.690209024" watchObservedRunningTime="2026-10-31 01:20:31.886518493 +0000 UTC m=+1403.693486050" Oct 31 01:20:32.066519 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:20:32.066485 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:20:32.707176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:32.707143 5756 generic.go:356] "Generic (PLEG): container finished" podID="55808545-23d5-4d0e-8d7f-f1364516c51b" containerID="60a3a8f0748c570664b9615411f386be346ec09ba911e4e92f338d87864ae5c8" exitCode=0 Oct 31 01:20:32.707582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:32.707226 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" event={"ID":"55808545-23d5-4d0e-8d7f-f1364516c51b","Type":"ContainerDied","Data":"60a3a8f0748c570664b9615411f386be346ec09ba911e4e92f338d87864ae5c8"} Oct 31 01:20:33.866202 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.866176 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:33.920881 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.920834 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca\") pod \"55808545-23d5-4d0e-8d7f-f1364516c51b\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " Oct 31 01:20:33.921064 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.921006 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m\" (UniqueName: \"kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m\") pod \"55808545-23d5-4d0e-8d7f-f1364516c51b\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " Oct 31 01:20:33.921064 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.921038 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca\") pod \"55808545-23d5-4d0e-8d7f-f1364516c51b\" (UID: \"55808545-23d5-4d0e-8d7f-f1364516c51b\") " Oct 31 01:20:33.921523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.921479 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca" pod "55808545-23d5-4d0e-8d7f-f1364516c51b" (UID: "55808545-23d5-4d0e-8d7f-f1364516c51b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:33.923517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.923481 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca" pod "55808545-23d5-4d0e-8d7f-f1364516c51b" (UID: "55808545-23d5-4d0e-8d7f-f1364516c51b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:33.923619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:33.923522 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m" pod "55808545-23d5-4d0e-8d7f-f1364516c51b" (UID: "55808545-23d5-4d0e-8d7f-f1364516c51b"). InnerVolumeSpecName "kube-api-access-krp6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:34.022891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:34.022788 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krp6m\" (UniqueName: \"kubernetes.io/projected/55808545-23d5-4d0e-8d7f-f1364516c51b-kube-api-access-krp6m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:34.022891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:34.022822 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:34.022891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:34.022881 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55808545-23d5-4d0e-8d7f-f1364516c51b-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:34.721756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:34.721668 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" Oct 31 01:20:34.721756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:34.721703 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln" event={"ID":"55808545-23d5-4d0e-8d7f-f1364516c51b","Type":"ContainerDied","Data":"679157006f76a94102df0b6e62c1c08f081d87488dea91293e9d85342fac7a66"} Oct 31 01:20:34.721954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:34.721767 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="679157006f76a94102df0b6e62c1c08f081d87488dea91293e9d85342fac7a66" Oct 31 01:20:37.052709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:37.052678 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c"] Oct 31 01:20:37.101568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:37.101525 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-300e228a-6nf8c"] Oct 31 01:20:38.311413 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:38.311379 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13991d9-47b7-4847-8638-6640dcef60d0" path="/var/lib/kubelet/pods/a13991d9-47b7-4847-8638-6640dcef60d0/volumes" Oct 31 01:20:42.068126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.068094 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw"] Oct 31 01:20:42.068944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.068923 5756 cpu_manager.go:414] "containerMap: removing container" podUID="55808545-23d5-4d0e-8d7f-f1364516c51b" containerName="revoke-keys" Oct 31 01:20:42.068944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.068942 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55808545-23d5-4d0e-8d7f-f1364516c51b" containerName="revoke-keys" Oct 31 01:20:42.069098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.069031 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="55808545-23d5-4d0e-8d7f-f1364516c51b" containerName="revoke-keys" Oct 31 01:20:42.073569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.073554 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125202 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125248 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125398 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125432 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67srm\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125472 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125573 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125643 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125746 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.125835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.125816 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.197248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.197216 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw"] Oct 31 01:20:42.227554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227514 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227593 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227627 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227722 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227770 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227913 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227966 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67srm\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.227988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.227974 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.228218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.228014 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.228218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.228052 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.228218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.228087 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.228480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.228437 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.228660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.228634 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.228819 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.228798 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.231074 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.231050 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.231491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.231466 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.289795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.289757 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.290036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.290017 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67srm\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.387730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.387653 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:42.450762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.449855 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:20:42.817022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.816998 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw"] Oct 31 01:20:42.818590 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:42.818568 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56ff08b_72d8_408f_8f13_008638ef60de.slice/crio-263ce7cc6b2977684c65ea592f799be0a4dabf84990730acd9700f423f857a29 WatchSource:0}: Error finding container 263ce7cc6b2977684c65ea592f799be0a4dabf84990730acd9700f423f857a29: Status 404 returned error can't find the container with id 263ce7cc6b2977684c65ea592f799be0a4dabf84990730acd9700f423f857a29 Oct 31 01:20:42.821115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.821080 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:20:42.821178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.821162 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:20:42.821219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:42.821206 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:20:43.299626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.299584 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn"] Oct 31 01:20:43.305300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.305272 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.371618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.371570 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn"] Oct 31 01:20:43.478925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.477663 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479175 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479346 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479333 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479445 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479596 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479688 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479836 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.479998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.479981 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.480138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.480125 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bps8c\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.581618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.581516 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.581618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.581585 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.581888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.581663 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.581888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.581714 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.581888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.581833 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.582079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.581913 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.582079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.582030 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.582079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.582075 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.582249 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.582180 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bps8c\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.589592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.583177 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.589592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.584914 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.589592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.585178 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.589592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.585565 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.589592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.586559 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.590746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.590724 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.591638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.591596 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.680527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.680491 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.687897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.686265 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bps8c\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:43.787120 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.785990 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" event={"ID":"c56ff08b-72d8-408f-8f13-008638ef60de","Type":"ContainerStarted","Data":"07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5"} Oct 31 01:20:43.787120 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.786060 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" event={"ID":"c56ff08b-72d8-408f-8f13-008638ef60de","Type":"ContainerStarted","Data":"263ce7cc6b2977684c65ea592f799be0a4dabf84990730acd9700f423f857a29"} Oct 31 01:20:43.925914 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:43.925610 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:44.388133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.388088 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:44.390367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.390317 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:44.390581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.390539 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:44.682090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.681983 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:44.865377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.865302 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podStartSLOduration=2.865283404 podStartE2EDuration="2.865283404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:43.911230347 +0000 UTC m=+1415.718197894" watchObservedRunningTime="2026-10-31 01:20:44.865283404 +0000 UTC m=+1416.672250948" Oct 31 01:20:44.882000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.881966 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88"] Oct 31 01:20:44.890490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.890463 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:44.989945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.973338 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:44.989945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:44.973508 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:20:45.054939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.050954 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 01:20:45.054939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.051365 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:45.182890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.157594 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57w9c\" (UniqueName: \"kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.255157 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.229050 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 31 01:20:45.275630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.275596 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.275931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.275910 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.279476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.279453 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88"] Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.359558 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn"] Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.408051 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.408135 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.408328 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57w9c\" (UniqueName: \"kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:45.410026 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e3ab4a_624d_4f34_a4ff_08b42e3e550d.slice/crio-db66c4abdd2bf00c914250212f868fcf1fee10f3a55a3d9a343e64a8d1adf08a WatchSource:0}: Error finding container db66c4abdd2bf00c914250212f868fcf1fee10f3a55a3d9a343e64a8d1adf08a: Status 404 returned error can't find the container with id db66c4abdd2bf00c914250212f868fcf1fee10f3a55a3d9a343e64a8d1adf08a Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.410089 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.410127 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.410775 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.445799 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.445955 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:20:45.466886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.446005 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:20:45.513956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.511184 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57w9c\" (UniqueName: \"kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.593364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.593299 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6380bd2-fzc88\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.669364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.669094 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:45.861535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.861462 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" event={"ID":"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d","Type":"ContainerStarted","Data":"db66c4abdd2bf00c914250212f868fcf1fee10f3a55a3d9a343e64a8d1adf08a"} Oct 31 01:20:45.930985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.930945 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:45.992972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:45.992900 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podStartSLOduration=2.992845995 podStartE2EDuration="2.992845995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:45.983332017 +0000 UTC m=+1417.790299560" watchObservedRunningTime="2026-10-31 01:20:45.992845995 +0000 UTC m=+1417.799813542" Oct 31 01:20:46.336957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.304077 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:46.336957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.304884 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:20:46.450920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.428881 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:46.450920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.428940 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:46.636727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.627351 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92"] Oct 31 01:20:46.708903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.707259 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92"] Oct 31 01:20:46.708903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.707415 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.783368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.770047 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"distinct-w0-26fbc4-kserve-self-signed-certs\"" Oct 31 01:20:46.828062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.828028 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 01:20:46.873655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.873608 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 31 01:20:46.887797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.887709 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.909679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.909645 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.910046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.910021 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szbwl\" (UniqueName: \"kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.910307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.910287 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.910491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.910472 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.910645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.910628 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:46.967889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.963299 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:46.967889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:46.963362 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.000449 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" event={"ID":"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d","Type":"ContainerStarted","Data":"fc088831af14fcad6042d69255b999fe9a145ee0c554bbc71021af22e901d10e"} Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.013672 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.013762 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szbwl\" (UniqueName: \"kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.013846 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.013922 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.013970 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.014022 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.022895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.022445 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.023417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.023062 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.037899 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.028931 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:20:47.037899 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.031563 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.061007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.060687 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.061007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.060956 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.109890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.109249 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szbwl\" (UniqueName: \"kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl\") pod \"distinct-w0-26fbc4-kserve-9855959d-jgm92\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.195430 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.183059 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88"] Oct 31 01:20:47.221635 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:47.220064 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713c3abe_806d_4293_8f3a_99c848ddc097.slice/crio-cfce82c65cf0147aa9e920294661079ba95f5bc291befa95b34e5a142f06b5ce WatchSource:0}: Error finding container cfce82c65cf0147aa9e920294661079ba95f5bc291befa95b34e5a142f06b5ce: Status 404 returned error can't find the container with id cfce82c65cf0147aa9e920294661079ba95f5bc291befa95b34e5a142f06b5ce Oct 31 01:20:47.299425 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.299383 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:47.391545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.390974 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:47.391545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.391059 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:47.990266 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:47.990229 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:48.011034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.010990 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:48.018892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.007854 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92"] Oct 31 01:20:48.124160 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.121335 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" event={"ID":"713c3abe-806d-4293-8f3a-99c848ddc097","Type":"ContainerStarted","Data":"cfce82c65cf0147aa9e920294661079ba95f5bc291befa95b34e5a142f06b5ce"} Oct 31 01:20:48.201001 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.200095 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" event={"ID":"92852b04-1bb0-4c84-8c7f-8ea8feec9f50","Type":"ContainerStarted","Data":"c35728f41ce2c9745097388c1d67e9ef30ce4050695da25121d4ad52501b4776"} Oct 31 01:20:48.394917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.389967 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:48.394917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.390025 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:48.927242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.927215 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:48.927414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:48.927263 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:49.140014 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.139977 5756 generic.go:356] "Generic (PLEG): container finished" podID="713c3abe-806d-4293-8f3a-99c848ddc097" containerID="fb34516e7f163b3ff5e4e5fc64f2597bbe4a176f79189d5d56be7e457e1bc630" exitCode=0 Oct 31 01:20:49.140649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.140625 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" event={"ID":"713c3abe-806d-4293-8f3a-99c848ddc097","Type":"ContainerDied","Data":"fb34516e7f163b3ff5e4e5fc64f2597bbe4a176f79189d5d56be7e457e1bc630"} Oct 31 01:20:49.144500 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.144473 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" event={"ID":"92852b04-1bb0-4c84-8c7f-8ea8feec9f50","Type":"ContainerStarted","Data":"9187859790d0f45e09bf7060c86104419d5d239edad97acbc8bc4e5b41a22fb7"} Oct 31 01:20:49.304363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.304331 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:49.388176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.388147 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:49.388366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.388199 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:49.507708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.496667 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" podStartSLOduration=3.496642092 podStartE2EDuration="3.496642092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:49.47465816 +0000 UTC m=+1421.281625704" watchObservedRunningTime="2026-10-31 01:20:49.496642092 +0000 UTC m=+1421.303609639" Oct 31 01:20:49.926801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.926763 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:49.926992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:49.926819 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:50.388283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.388215 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:50.388283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.388266 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:50.419948 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.419924 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:50.495616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.495507 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c\" (UniqueName: \"kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c\") pod \"713c3abe-806d-4293-8f3a-99c848ddc097\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " Oct 31 01:20:50.495616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.495591 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca\") pod \"713c3abe-806d-4293-8f3a-99c848ddc097\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " Oct 31 01:20:50.495853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.495736 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca\") pod \"713c3abe-806d-4293-8f3a-99c848ddc097\" (UID: \"713c3abe-806d-4293-8f3a-99c848ddc097\") " Oct 31 01:20:50.496241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.496209 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca" pod "713c3abe-806d-4293-8f3a-99c848ddc097" (UID: "713c3abe-806d-4293-8f3a-99c848ddc097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:50.496521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.496504 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:50.497834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.497808 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca" pod "713c3abe-806d-4293-8f3a-99c848ddc097" (UID: "713c3abe-806d-4293-8f3a-99c848ddc097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:50.498112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.498094 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c" pod "713c3abe-806d-4293-8f3a-99c848ddc097" (UID: "713c3abe-806d-4293-8f3a-99c848ddc097"). InnerVolumeSpecName "kube-api-access-57w9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:50.598199 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.598157 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57w9c\" (UniqueName: \"kubernetes.io/projected/713c3abe-806d-4293-8f3a-99c848ddc097-kube-api-access-57w9c\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:50.598199 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.598188 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/713c3abe-806d-4293-8f3a-99c848ddc097-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:50.927168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.927131 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:50.927347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:50.927195 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:51.195104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.195028 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" Oct 31 01:20:51.195325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.195123 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88" event={"ID":"713c3abe-806d-4293-8f3a-99c848ddc097","Type":"ContainerDied","Data":"cfce82c65cf0147aa9e920294661079ba95f5bc291befa95b34e5a142f06b5ce"} Oct 31 01:20:51.195325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.195160 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfce82c65cf0147aa9e920294661079ba95f5bc291befa95b34e5a142f06b5ce" Oct 31 01:20:51.388356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.388315 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 01:20:51.388758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.388376 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 01:20:51.927323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.927289 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:51.927507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:51.927346 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:52.884828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:52.884796 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:52.886190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:52.886146 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:52.887082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:52.887066 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:20:52.927407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:52.927378 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:20:52.927579 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:52.927429 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 31 01:20:54.084073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.084036 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s"] Oct 31 01:20:54.084964 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.084943 5756 cpu_manager.go:414] "containerMap: removing container" podUID="713c3abe-806d-4293-8f3a-99c848ddc097" containerName="revoke-keys" Oct 31 01:20:54.085013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.084967 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713c3abe-806d-4293-8f3a-99c848ddc097" containerName="revoke-keys" Oct 31 01:20:54.085196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.085182 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="713c3abe-806d-4293-8f3a-99c848ddc097" containerName="revoke-keys" Oct 31 01:20:54.089150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.089128 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.145882 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.145833 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd8nm\" (UniqueName: \"kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.146064 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.145893 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.146064 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.146031 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.199088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.199044 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:20:54.199381 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.199273 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:20:54.204467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.204439 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s"] Oct 31 01:20:54.247900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.247844 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd8nm\" (UniqueName: \"kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.248049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.247916 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.248109 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.248050 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.248660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.248639 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.250528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.250507 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.290270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.290238 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:54.291570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.291546 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:54.294259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.294237 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:20:54.312552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.312526 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd8nm\" (UniqueName: \"kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm\") pod \"maas-api-revoke-sub-8f829c1e-7qn6s\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.406765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.406685 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:54.805422 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:54.805385 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s"] Oct 31 01:20:54.806612 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:20:54.806585 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcc976d_41bb_4a4e_b5c5_3d78712c511f.slice/crio-2ddb1c0f33eb7a5e06966d8f8d640a7f83d51305aa3ef5bc411738d76c81727d WatchSource:0}: Error finding container 2ddb1c0f33eb7a5e06966d8f8d640a7f83d51305aa3ef5bc411738d76c81727d: Status 404 returned error can't find the container with id 2ddb1c0f33eb7a5e06966d8f8d640a7f83d51305aa3ef5bc411738d76c81727d Oct 31 01:20:55.228346 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:55.228302 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" event={"ID":"1dcc976d-41bb-4a4e-b5c5-3d78712c511f","Type":"ContainerStarted","Data":"0d8d8ad7cd408d72c39120d23bad8234b64a6040de63416b25724dc7a58a89f3"} Oct 31 01:20:55.228822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:55.228360 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" event={"ID":"1dcc976d-41bb-4a4e-b5c5-3d78712c511f","Type":"ContainerStarted","Data":"2ddb1c0f33eb7a5e06966d8f8d640a7f83d51305aa3ef5bc411738d76c81727d"} Oct 31 01:20:55.354468 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:55.354412 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" podStartSLOduration=1.354397359 podStartE2EDuration="1.354397359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:55.350842705 +0000 UTC m=+1427.157810248" watchObservedRunningTime="2026-10-31 01:20:55.354397359 +0000 UTC m=+1427.161364901" Oct 31 01:20:56.236335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:56.236301 5756 generic.go:356] "Generic (PLEG): container finished" podID="1dcc976d-41bb-4a4e-b5c5-3d78712c511f" containerID="0d8d8ad7cd408d72c39120d23bad8234b64a6040de63416b25724dc7a58a89f3" exitCode=0 Oct 31 01:20:56.236716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:56.236382 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" event={"ID":"1dcc976d-41bb-4a4e-b5c5-3d78712c511f","Type":"ContainerDied","Data":"0d8d8ad7cd408d72c39120d23bad8234b64a6040de63416b25724dc7a58a89f3"} Oct 31 01:20:57.304376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.304344 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:57.324797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.324769 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:20:57.389687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.389664 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:57.496477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.496390 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm\" (UniqueName: \"kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm\") pod \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " Oct 31 01:20:57.496477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.496450 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca\") pod \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " Oct 31 01:20:57.496712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.496682 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca\") pod \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\" (UID: \"1dcc976d-41bb-4a4e-b5c5-3d78712c511f\") " Oct 31 01:20:57.496926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.496904 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca" pod "1dcc976d-41bb-4a4e-b5c5-3d78712c511f" (UID: "1dcc976d-41bb-4a4e-b5c5-3d78712c511f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:57.497182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.497163 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:57.498649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.498620 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm" pod "1dcc976d-41bb-4a4e-b5c5-3d78712c511f" (UID: "1dcc976d-41bb-4a4e-b5c5-3d78712c511f"). InnerVolumeSpecName "kube-api-access-cd8nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:57.498748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.498732 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca" pod "1dcc976d-41bb-4a4e-b5c5-3d78712c511f" (UID: "1dcc976d-41bb-4a4e-b5c5-3d78712c511f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:57.599009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.598980 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd8nm\" (UniqueName: \"kubernetes.io/projected/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-kube-api-access-cd8nm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:57.599009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:57.599008 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dcc976d-41bb-4a4e-b5c5-3d78712c511f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:20:58.257965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:58.257922 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" Oct 31 01:20:58.260780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:58.260745 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s" event={"ID":"1dcc976d-41bb-4a4e-b5c5-3d78712c511f","Type":"ContainerDied","Data":"2ddb1c0f33eb7a5e06966d8f8d640a7f83d51305aa3ef5bc411738d76c81727d"} Oct 31 01:20:58.261016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:58.260826 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ddb1c0f33eb7a5e06966d8f8d640a7f83d51305aa3ef5bc411738d76c81727d" Oct 31 01:20:58.277075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:20:58.277049 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 01:21:01.230890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:01.223331 5756 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:21:01.269919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:01.268954 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:21:01.366889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:01.360008 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:21:01.683890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:01.682966 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:21:01.851885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:01.850989 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:21:02.025621 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:02.024650 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 31 01:21:02.105538 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:21:02.105496 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:21:04.915656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.915627 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf"] Oct 31 01:21:04.916216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.916202 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1dcc976d-41bb-4a4e-b5c5-3d78712c511f" containerName="revoke-keys" Oct 31 01:21:04.916258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.916217 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcc976d-41bb-4a4e-b5c5-3d78712c511f" containerName="revoke-keys" Oct 31 01:21:04.916324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.916314 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcc976d-41bb-4a4e-b5c5-3d78712c511f" containerName="revoke-keys" Oct 31 01:21:04.919998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.919979 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:04.974629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.974596 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:04.974775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:04.974634 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:21:05.029080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.029049 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf"] Oct 31 01:21:05.066364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.066331 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.066565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.066508 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cfv\" (UniqueName: \"kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.066637 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.066578 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.168177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.168093 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.168334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.168199 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cfv\" (UniqueName: \"kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.168334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.168222 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.168928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.168902 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.170726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.170707 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.229248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.229213 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cfv\" (UniqueName: \"kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv\") pod \"maas-api-revoke-sub-fe762391-trxqf\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.234069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.234045 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:05.625897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:05.625844 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf"] Oct 31 01:21:05.630124 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:21:05.630090 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c6b9b8_5c31_499b_a2ec_69b8a6e421f1.slice/crio-85f394491d03fd059a7f8e53445c0d1ec728b42ec0bfc66357186066f098cfc6 WatchSource:0}: Error finding container 85f394491d03fd059a7f8e53445c0d1ec728b42ec0bfc66357186066f098cfc6: Status 404 returned error can't find the container with id 85f394491d03fd059a7f8e53445c0d1ec728b42ec0bfc66357186066f098cfc6 Oct 31 01:21:06.385304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:06.385272 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" event={"ID":"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1","Type":"ContainerStarted","Data":"08f332103549786e711287db222ccac79a5ea12b304af3ef1ac7850e53918d5e"} Oct 31 01:21:06.385713 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:06.385330 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" event={"ID":"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1","Type":"ContainerStarted","Data":"85f394491d03fd059a7f8e53445c0d1ec728b42ec0bfc66357186066f098cfc6"} Oct 31 01:21:06.507449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:06.507398 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" podStartSLOduration=2.50738501 podStartE2EDuration="2.50738501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:06.506073247 +0000 UTC m=+1438.313040789" watchObservedRunningTime="2026-10-31 01:21:06.50738501 +0000 UTC m=+1438.314352553" Oct 31 01:21:07.393846 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:07.393812 5756 generic.go:356] "Generic (PLEG): container finished" podID="71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" containerID="08f332103549786e711287db222ccac79a5ea12b304af3ef1ac7850e53918d5e" exitCode=0 Oct 31 01:21:07.394261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:07.393909 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" event={"ID":"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1","Type":"ContainerDied","Data":"08f332103549786e711287db222ccac79a5ea12b304af3ef1ac7850e53918d5e"} Oct 31 01:21:08.567541 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.567517 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:08.609387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.609332 5756 scope.go:122] "RemoveContainer" containerID="4ccc1e4375d7032c6e8410556e8908de3d7491afc39c126a1a55341798ae3859" Oct 31 01:21:08.621103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.621079 5756 scope.go:122] "RemoveContainer" containerID="9dd71d5c1e4997df8348e933404fa0bd977e8d92f3c42bf0b61ba414984d923c" Oct 31 01:21:08.630181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.630164 5756 scope.go:122] "RemoveContainer" containerID="3c7c82e9830b42dfcae621602cefcd0b25505cee386109dbd738bbf448fd351a" Oct 31 01:21:08.638905 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.638875 5756 scope.go:122] "RemoveContainer" containerID="1db937c31b1523ad8d11a3eeb1baba228c9e7943a6354555e7ece69b7baf6341" Oct 31 01:21:08.647238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.647205 5756 scope.go:122] "RemoveContainer" containerID="8999be19509a177460f683f88c5ee14f506799576bff5dc7c7801bdb227b2f86" Oct 31 01:21:08.716570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.716524 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv\" (UniqueName: \"kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv\") pod \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " Oct 31 01:21:08.716752 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.716629 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca\") pod \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " Oct 31 01:21:08.716822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.716774 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca\") pod \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\" (UID: \"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1\") " Oct 31 01:21:08.717078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.717052 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca" pod "71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" (UID: "71c6b9b8-5c31-499b-a2ec-69b8a6e421f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:08.717386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.717339 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:21:08.719189 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.719161 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca" pod "71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" (UID: "71c6b9b8-5c31-499b-a2ec-69b8a6e421f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:08.719486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.719465 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv" pod "71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" (UID: "71c6b9b8-5c31-499b-a2ec-69b8a6e421f1"). InnerVolumeSpecName "kube-api-access-n4cfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:08.818195 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.818168 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cfv\" (UniqueName: \"kubernetes.io/projected/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-kube-api-access-n4cfv\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:21:08.818195 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:08.818193 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:21:09.409512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:09.409482 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" Oct 31 01:21:09.409703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:09.409485 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf" event={"ID":"71c6b9b8-5c31-499b-a2ec-69b8a6e421f1","Type":"ContainerDied","Data":"85f394491d03fd059a7f8e53445c0d1ec728b42ec0bfc66357186066f098cfc6"} Oct 31 01:21:09.409703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:09.409605 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f394491d03fd059a7f8e53445c0d1ec728b42ec0bfc66357186066f098cfc6" Oct 31 01:21:24.079055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:24.078973 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:21:26.870985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:26.870942 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:21:27.163003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.162912 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2"] Oct 31 01:21:27.164100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.164072 5756 cpu_manager.go:414] "containerMap: removing container" podUID="71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" containerName="revoke-keys" Oct 31 01:21:27.164270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.164254 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" containerName="revoke-keys" Oct 31 01:21:27.164551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.164535 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" containerName="revoke-keys" Oct 31 01:21:27.169012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.168991 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.226362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.216405 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2"] Oct 31 01:21:27.236922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.236543 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwxh7\" (UniqueName: \"kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.236922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.236632 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.236922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.236670 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.265387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.265358 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:21:27.265699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.265681 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:21:27.368765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.368723 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwxh7\" (UniqueName: \"kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.369134 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.369109 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.369289 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.369273 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.371093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.371067 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.380338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.380313 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.549470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.543049 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwxh7\" (UniqueName: \"kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7\") pod \"maas-api-revoke-sub-466a8c0c-vsnc2\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:27.594572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:27.591513 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:28.024802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:28.024770 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2"] Oct 31 01:21:28.605683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:28.605604 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" event={"ID":"1d060506-e764-4151-a9f2-220e058e7f4c","Type":"ContainerStarted","Data":"11cc66591ca179bbde99634f879c33a2046b511741b6ab4dc29422b5899425c5"} Oct 31 01:21:28.605683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:28.605669 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" event={"ID":"1d060506-e764-4151-a9f2-220e058e7f4c","Type":"ContainerStarted","Data":"bf527890281b232314e91fe8680240cf6e8b9f4d7755e2b75f8dce731d0202e1"} Oct 31 01:21:28.724643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:28.724594 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" podStartSLOduration=1.7245799590000002 podStartE2EDuration="1.724579959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:28.720766851 +0000 UTC m=+1460.527734390" watchObservedRunningTime="2026-10-31 01:21:28.724579959 +0000 UTC m=+1460.531547501" Oct 31 01:21:29.613587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:29.613554 5756 generic.go:356] "Generic (PLEG): container finished" podID="1d060506-e764-4151-a9f2-220e058e7f4c" containerID="11cc66591ca179bbde99634f879c33a2046b511741b6ab4dc29422b5899425c5" exitCode=0 Oct 31 01:21:29.614069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:29.613615 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" event={"ID":"1d060506-e764-4151-a9f2-220e058e7f4c","Type":"ContainerDied","Data":"11cc66591ca179bbde99634f879c33a2046b511741b6ab4dc29422b5899425c5"} Oct 31 01:21:30.778346 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.778320 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:30.843304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.843270 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca\") pod \"1d060506-e764-4151-a9f2-220e058e7f4c\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " Oct 31 01:21:30.843304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.843309 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca\") pod \"1d060506-e764-4151-a9f2-220e058e7f4c\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " Oct 31 01:21:30.843639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.843618 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca" pod "1d060506-e764-4151-a9f2-220e058e7f4c" (UID: "1d060506-e764-4151-a9f2-220e058e7f4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:30.843715 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.843702 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:21:30.845526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.845497 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca" pod "1d060506-e764-4151-a9f2-220e058e7f4c" (UID: "1d060506-e764-4151-a9f2-220e058e7f4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:30.944985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.944895 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7\" (UniqueName: \"kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7\") pod \"1d060506-e764-4151-a9f2-220e058e7f4c\" (UID: \"1d060506-e764-4151-a9f2-220e058e7f4c\") " Oct 31 01:21:30.945412 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.945390 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d060506-e764-4151-a9f2-220e058e7f4c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:21:30.947047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:30.947024 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7" pod "1d060506-e764-4151-a9f2-220e058e7f4c" (UID: "1d060506-e764-4151-a9f2-220e058e7f4c"). InnerVolumeSpecName "kube-api-access-nwxh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:31.046896 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:31.046841 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwxh7\" (UniqueName: \"kubernetes.io/projected/1d060506-e764-4151-a9f2-220e058e7f4c-kube-api-access-nwxh7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:21:31.627408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:31.627378 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" Oct 31 01:21:31.627582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:31.627378 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2" event={"ID":"1d060506-e764-4151-a9f2-220e058e7f4c","Type":"ContainerDied","Data":"bf527890281b232314e91fe8680240cf6e8b9f4d7755e2b75f8dce731d0202e1"} Oct 31 01:21:31.627582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:31.627501 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf527890281b232314e91fe8680240cf6e8b9f4d7755e2b75f8dce731d0202e1" Oct 31 01:21:32.067075 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:21:32.067028 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:21:46.244281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.244245 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q"] Oct 31 01:21:46.244809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.244795 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1d060506-e764-4151-a9f2-220e058e7f4c" containerName="revoke-keys" Oct 31 01:21:46.244852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.244810 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d060506-e764-4151-a9f2-220e058e7f4c" containerName="revoke-keys" Oct 31 01:21:46.244926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.244916 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d060506-e764-4151-a9f2-220e058e7f4c" containerName="revoke-keys" Oct 31 01:21:46.248515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.248497 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.350005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.349977 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-26fbc4\"/\"distinct-2-w0-26fbc4-kserve-self-signed-certs\"" Oct 31 01:21:46.350578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.350554 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.350647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.350594 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.350647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.350629 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.350846 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.350828 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj9c4\" (UniqueName: \"kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.350913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.350895 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.351035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.351014 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.353775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.353753 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q"] Oct 31 01:21:46.452421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.452379 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj9c4\" (UniqueName: \"kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.452735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.452713 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.453025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.452981 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.453426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.453396 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.453603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.453587 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.453774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.453759 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.454166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.454138 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.454264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.453487 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.455323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.455299 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.458042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.458001 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.478773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.478741 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.509831 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.509758 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj9c4\" (UniqueName: \"kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4\") pod \"distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.560503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.560467 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:46.995560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:46.995540 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q"] Oct 31 01:21:46.996492 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:21:46.996463 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99c1d371_db3f_4b9f_b4e5_d2a709b4486a.slice/crio-1169ae85c5b384487c59b2b6f70dfd425beb27c249fa731cbf32a2853a1c7b69 WatchSource:0}: Error finding container 1169ae85c5b384487c59b2b6f70dfd425beb27c249fa731cbf32a2853a1c7b69: Status 404 returned error can't find the container with id 1169ae85c5b384487c59b2b6f70dfd425beb27c249fa731cbf32a2853a1c7b69 Oct 31 01:21:47.743909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:47.743840 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" event={"ID":"99c1d371-db3f-4b9f-b4e5-d2a709b4486a","Type":"ContainerStarted","Data":"762cb180d117120ea671fa3284a808202c708ccf0ccfdcaadc9a8fe336868a52"} Oct 31 01:21:47.744378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:47.743937 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" event={"ID":"99c1d371-db3f-4b9f-b4e5-d2a709b4486a","Type":"ContainerStarted","Data":"1169ae85c5b384487c59b2b6f70dfd425beb27c249fa731cbf32a2853a1c7b69"} Oct 31 01:21:47.860016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:47.859953 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" podStartSLOduration=1.8599381560000001 podStartE2EDuration="1.859938156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:47.858548857 +0000 UTC m=+1479.665516402" watchObservedRunningTime="2026-10-31 01:21:47.859938156 +0000 UTC m=+1479.666905728" Oct 31 01:21:48.561199 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:48.561110 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:56.560883 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:56.560829 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:56.573724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:56.573699 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:21:56.864002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:21:56.863926 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 01:22:02.066399 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:22:02.066359 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:22:04.314936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.314903 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs"] Oct 31 01:22:04.320446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.320425 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.373528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.373500 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs"] Oct 31 01:22:04.418004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.417976 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:04.418169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.418030 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:22:04.422692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.422671 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gld65\" (UniqueName: \"kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.422950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.422932 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.423066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.423049 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.524162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.524128 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.524328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.524191 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.524328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.524255 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gld65\" (UniqueName: \"kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.524771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.524748 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.526724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.526702 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.581630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.581559 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gld65\" (UniqueName: \"kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65\") pod \"maas-api-revoke-sub-6450408c-mpzzs\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:04.637020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:04.636996 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:05.016583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:05.016551 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs"] Oct 31 01:22:05.017667 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:22:05.017642 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4434f2_0759_4f3d_adaf_20a0ed8d37c7.slice/crio-d7c17c0cac8c3e72c4d3e0e761b99de77e20f1e3a5c96ed0ccf9c85b301f4a53 WatchSource:0}: Error finding container d7c17c0cac8c3e72c4d3e0e761b99de77e20f1e3a5c96ed0ccf9c85b301f4a53: Status 404 returned error can't find the container with id d7c17c0cac8c3e72c4d3e0e761b99de77e20f1e3a5c96ed0ccf9c85b301f4a53 Oct 31 01:22:05.912740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:05.912544 5756 generic.go:356] "Generic (PLEG): container finished" podID="8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" containerID="1efd019ec67e92a8b7f4c9feb8f0450e5ce66b61b1f40414cfeca731fa612b66" exitCode=0 Oct 31 01:22:05.912740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:05.912596 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" event={"ID":"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7","Type":"ContainerDied","Data":"1efd019ec67e92a8b7f4c9feb8f0450e5ce66b61b1f40414cfeca731fa612b66"} Oct 31 01:22:05.912740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:05.912699 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" event={"ID":"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7","Type":"ContainerStarted","Data":"d7c17c0cac8c3e72c4d3e0e761b99de77e20f1e3a5c96ed0ccf9c85b301f4a53"} Oct 31 01:22:07.072352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.072329 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:07.158578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.158547 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65\" (UniqueName: \"kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65\") pod \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " Oct 31 01:22:07.158747 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.158606 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca\") pod \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " Oct 31 01:22:07.160943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.160914 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65" pod "8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" (UID: "8d4434f2-0759-4f3d-adaf-20a0ed8d37c7"). InnerVolumeSpecName "kube-api-access-gld65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:07.160943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.160935 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca" pod "8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" (UID: "8d4434f2-0759-4f3d-adaf-20a0ed8d37c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:07.259698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.259626 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca\") pod \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\" (UID: \"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7\") " Oct 31 01:22:07.260006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.259984 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca" pod "8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" (UID: "8d4434f2-0759-4f3d-adaf-20a0ed8d37c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:07.260068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.260021 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gld65\" (UniqueName: \"kubernetes.io/projected/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-kube-api-access-gld65\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:22:07.260068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.260040 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:22:07.360909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.360846 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:22:07.928119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.928089 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" Oct 31 01:22:07.928307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.928128 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs" event={"ID":"8d4434f2-0759-4f3d-adaf-20a0ed8d37c7","Type":"ContainerDied","Data":"d7c17c0cac8c3e72c4d3e0e761b99de77e20f1e3a5c96ed0ccf9c85b301f4a53"} Oct 31 01:22:07.928307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:07.928180 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7c17c0cac8c3e72c4d3e0e761b99de77e20f1e3a5c96ed0ccf9c85b301f4a53" Oct 31 01:22:08.694896 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:08.694849 5756 scope.go:122] "RemoveContainer" containerID="287b30c4440393661f12bf99582c34af5c5cccb364fcfbb88106b8068da18af4" Oct 31 01:22:08.705207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:08.705183 5756 scope.go:122] "RemoveContainer" containerID="cd8efeb6155dddb16e03c6c50325b47e725219b5f955c9f278978dd1fb546071" Oct 31 01:22:21.526764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:21.526732 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:22:32.067398 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:22:32.067356 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:22:35.628278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.628238 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh"] Oct 31 01:22:35.630021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.629996 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" containerName="revoke-keys" Oct 31 01:22:35.630249 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.630234 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" containerName="revoke-keys" Oct 31 01:22:35.630625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.630609 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" containerName="revoke-keys" Oct 31 01:22:35.636224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.636200 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.747327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.747293 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f259852e\"/\"kube-root-ca.crt\"" Oct 31 01:22:35.747668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.747651 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f259852e\"/\"openshift-service-ca.crt\"" Oct 31 01:22:35.747910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.747851 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f259852e\"/\"e2e-lifecycle-model-f259852e-kserve-self-signed-certs\"" Oct 31 01:22:35.748031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.747922 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f259852e\"/\"default-dockercfg-wt8vw\"" Oct 31 01:22:35.749273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.749249 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh"] Oct 31 01:22:35.757704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.757679 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.757816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.757796 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.757885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.757853 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.758025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.758010 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.758095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.758078 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66vr7\" (UniqueName: \"kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.758151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.758134 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.859737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.859695 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.859973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.859777 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66vr7\" (UniqueName: \"kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.859973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.859822 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.859973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.859895 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.860168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.859999 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.860168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.860054 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.861648 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.861625 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.862259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.862233 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.862516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.862491 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.865153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.865124 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.870398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.870376 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.918248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.918169 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66vr7\" (UniqueName: \"kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7\") pod \"e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:35.982481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:35.982440 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:36.370321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:36.370234 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh"] Oct 31 01:22:37.143359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:37.143318 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" event={"ID":"e0382435-c90b-452c-a93f-30d2741c65d8","Type":"ContainerStarted","Data":"a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a"} Oct 31 01:22:37.143793 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:37.143377 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" event={"ID":"e0382435-c90b-452c-a93f-30d2741c65d8","Type":"ContainerStarted","Data":"affbfecf46572f7bebd4fc23f12f79764e535759159567cace79c238164a691f"} Oct 31 01:22:37.257616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:37.257547 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" podStartSLOduration=2.257531776 podStartE2EDuration="2.257531776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:37.255976514 +0000 UTC m=+1529.062944059" watchObservedRunningTime="2026-10-31 01:22:37.257531776 +0000 UTC m=+1529.064499361" Oct 31 01:22:37.982942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:37.982895 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:45.285890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.284904 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f"] Oct 31 01:22:45.305892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.304108 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.362021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.361986 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f"] Oct 31 01:22:45.386044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386004 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386062 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386258 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386335 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m247z\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386375 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386615 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386652 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386691 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.386810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.386734 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.487771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487690 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.487771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487769 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m247z\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487796 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487851 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487894 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487922 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.487951 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.488056 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.488092 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.488302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.488157 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.494520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.488303 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.494520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.488823 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.494520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.488884 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.494520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.489131 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.494520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.490704 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.494520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.491712 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.544302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.544232 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.544433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.544393 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m247z\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.622355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.622321 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:45.983673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.983600 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:45.996124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:45.996090 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:46.091973 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:22:46.091940 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0429a525_32e3_45a1_aecd_829241e2ebd8.slice/crio-0a134f9054ea4de5e9eaed56d161be5b587c543292f3b1046980932552c5aef0 WatchSource:0}: Error finding container 0a134f9054ea4de5e9eaed56d161be5b587c543292f3b1046980932552c5aef0: Status 404 returned error can't find the container with id 0a134f9054ea4de5e9eaed56d161be5b587c543292f3b1046980932552c5aef0 Oct 31 01:22:46.094307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.094275 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:22:46.094393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.094369 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:22:46.094433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.094403 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:22:46.102609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.102588 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f"] Oct 31 01:22:46.203706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.203669 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" event={"ID":"0429a525-32e3-45a1-aecd-829241e2ebd8","Type":"ContainerStarted","Data":"18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15"} Oct 31 01:22:46.203931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.203739 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" event={"ID":"0429a525-32e3-45a1-aecd-829241e2ebd8","Type":"ContainerStarted","Data":"0a134f9054ea4de5e9eaed56d161be5b587c543292f3b1046980932552c5aef0"} Oct 31 01:22:46.220514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.220492 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:22:46.323924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.323834 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podStartSLOduration=1.323820377 podStartE2EDuration="1.323820377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:46.321433775 +0000 UTC m=+1538.128401316" watchObservedRunningTime="2026-10-31 01:22:46.323820377 +0000 UTC m=+1538.130787993" Oct 31 01:22:46.623361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:46.623290 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:47.624736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:47.624696 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:22:47.625186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:47.624761 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 01:22:48.622696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:48.622664 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:22:48.622878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:48.622747 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 01:22:49.623606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:49.623572 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:22:49.624006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:49.623626 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 01:22:50.622659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:50.622627 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:22:50.622834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:50.622689 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 01:22:51.623511 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:51.623479 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:22:51.623927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:51.623539 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 01:22:52.623532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:52.623492 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 31 01:22:52.624023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:52.623550 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 31 01:22:53.627434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:53.627400 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:53.627942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:53.627795 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:53.628455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:53.628433 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:22:56.944687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:56.944655 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r"] Oct 31 01:22:56.948568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:56.948550 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:56.970236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:56.970207 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:56.970434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:56.970413 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:56.970554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:56.970533 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgclk\" (UniqueName: \"kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.000727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.000696 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:22:57.050258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.050233 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:22:57.053700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.053678 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r"] Oct 31 01:22:57.071427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.071393 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.071597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.071474 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgclk\" (UniqueName: \"kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.071597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.071522 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.072167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.072147 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.073951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.073930 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.128172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.128145 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgclk\" (UniqueName: \"kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk\") pod \"maas-api-revoke-sub-33e221b4-tq54r\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.263480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.263409 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:22:57.659969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:57.659937 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r"] Oct 31 01:22:57.660421 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:22:57.660399 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06dd1f99_c151_4b46_9853_eab6054ef5da.slice/crio-2a09ea97836824e73520c1a3721d73340d6dd6bbe5d12fc969260f186862b615 WatchSource:0}: Error finding container 2a09ea97836824e73520c1a3721d73340d6dd6bbe5d12fc969260f186862b615: Status 404 returned error can't find the container with id 2a09ea97836824e73520c1a3721d73340d6dd6bbe5d12fc969260f186862b615 Oct 31 01:22:58.325587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:58.325554 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" event={"ID":"06dd1f99-c151-4b46-9853-eab6054ef5da","Type":"ContainerStarted","Data":"9d5ed4c7380f40e8384481cba8537098cb946cbd72b02e1333f6ed101d994f9f"} Oct 31 01:22:58.326002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:58.325611 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" event={"ID":"06dd1f99-c151-4b46-9853-eab6054ef5da","Type":"ContainerStarted","Data":"2a09ea97836824e73520c1a3721d73340d6dd6bbe5d12fc969260f186862b615"} Oct 31 01:22:58.441277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:58.441226 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" podStartSLOduration=2.441212817 podStartE2EDuration="2.441212817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:58.439513487 +0000 UTC m=+1550.246481032" watchObservedRunningTime="2026-10-31 01:22:58.441212817 +0000 UTC m=+1550.248180359" Oct 31 01:22:59.334341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:59.334309 5756 generic.go:356] "Generic (PLEG): container finished" podID="06dd1f99-c151-4b46-9853-eab6054ef5da" containerID="9d5ed4c7380f40e8384481cba8537098cb946cbd72b02e1333f6ed101d994f9f" exitCode=0 Oct 31 01:22:59.334770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:22:59.334389 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" event={"ID":"06dd1f99-c151-4b46-9853-eab6054ef5da","Type":"ContainerDied","Data":"9d5ed4c7380f40e8384481cba8537098cb946cbd72b02e1333f6ed101d994f9f"} Oct 31 01:23:00.491657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.491630 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:23:00.624229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.624139 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca\") pod \"06dd1f99-c151-4b46-9853-eab6054ef5da\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " Oct 31 01:23:00.624401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.624230 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk\" (UniqueName: \"kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk\") pod \"06dd1f99-c151-4b46-9853-eab6054ef5da\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " Oct 31 01:23:00.624401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.624283 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca\") pod \"06dd1f99-c151-4b46-9853-eab6054ef5da\" (UID: \"06dd1f99-c151-4b46-9853-eab6054ef5da\") " Oct 31 01:23:00.624724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.624694 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca" pod "06dd1f99-c151-4b46-9853-eab6054ef5da" (UID: "06dd1f99-c151-4b46-9853-eab6054ef5da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:00.625109 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.625082 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:23:00.626527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.626497 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca" pod "06dd1f99-c151-4b46-9853-eab6054ef5da" (UID: "06dd1f99-c151-4b46-9853-eab6054ef5da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:00.626632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.626593 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk" pod "06dd1f99-c151-4b46-9853-eab6054ef5da" (UID: "06dd1f99-c151-4b46-9853-eab6054ef5da"). InnerVolumeSpecName "kube-api-access-dgclk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:00.726532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.726497 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06dd1f99-c151-4b46-9853-eab6054ef5da-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:23:00.726532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:00.726530 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgclk\" (UniqueName: \"kubernetes.io/projected/06dd1f99-c151-4b46-9853-eab6054ef5da-kube-api-access-dgclk\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:23:01.348702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:01.348671 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" Oct 31 01:23:01.348702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:01.348686 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r" event={"ID":"06dd1f99-c151-4b46-9853-eab6054ef5da","Type":"ContainerDied","Data":"2a09ea97836824e73520c1a3721d73340d6dd6bbe5d12fc969260f186862b615"} Oct 31 01:23:01.348950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:01.348732 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a09ea97836824e73520c1a3721d73340d6dd6bbe5d12fc969260f186862b615" Oct 31 01:23:02.067208 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:23:02.067167 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:23:05.721607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:05.721571 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:23:08.548756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.548722 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf"] Oct 31 01:23:08.549307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.549292 5756 cpu_manager.go:414] "containerMap: removing container" podUID="06dd1f99-c151-4b46-9853-eab6054ef5da" containerName="revoke-keys" Oct 31 01:23:08.549351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.549309 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06dd1f99-c151-4b46-9853-eab6054ef5da" containerName="revoke-keys" Oct 31 01:23:08.549446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.549431 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="06dd1f99-c151-4b46-9853-eab6054ef5da" containerName="revoke-keys" Oct 31 01:23:08.553105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.553089 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.609069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.609041 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf"] Oct 31 01:23:08.653366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.653337 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:23:08.653546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.653415 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:23:08.742189 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.742160 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzksb\" (UniqueName: \"kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.742364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.742215 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.742442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.742421 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.843526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.843443 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzksb\" (UniqueName: \"kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.843699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.843522 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.843699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.843667 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.844340 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.844317 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.846138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.846118 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:08.899977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:08.899940 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzksb\" (UniqueName: \"kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb\") pod \"maas-api-revoke-sub-a1383412-njklf\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:09.166899 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:09.166788 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:09.563024 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:09.562992 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf"] Oct 31 01:23:09.564062 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:23:09.564032 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aac2a03_80c5_4f70_b09b_c72312711a86.slice/crio-2e0d5fb42c22940eef6afcb1c50c9924d49a3a933a7f08812e225712e7b6e732 WatchSource:0}: Error finding container 2e0d5fb42c22940eef6afcb1c50c9924d49a3a933a7f08812e225712e7b6e732: Status 404 returned error can't find the container with id 2e0d5fb42c22940eef6afcb1c50c9924d49a3a933a7f08812e225712e7b6e732 Oct 31 01:23:10.412647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:10.412613 5756 generic.go:356] "Generic (PLEG): container finished" podID="9aac2a03-80c5-4f70-b09b-c72312711a86" containerID="879c174794f3fe3f3a7794876836d706de0f69ab59db92542ef0a497db32fe3c" exitCode=0 Oct 31 01:23:10.412806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:10.412705 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" event={"ID":"9aac2a03-80c5-4f70-b09b-c72312711a86","Type":"ContainerDied","Data":"879c174794f3fe3f3a7794876836d706de0f69ab59db92542ef0a497db32fe3c"} Oct 31 01:23:10.412806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:10.412769 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" event={"ID":"9aac2a03-80c5-4f70-b09b-c72312711a86","Type":"ContainerStarted","Data":"2e0d5fb42c22940eef6afcb1c50c9924d49a3a933a7f08812e225712e7b6e732"} Oct 31 01:23:11.581416 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.581394 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:11.679310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.679226 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca\") pod \"9aac2a03-80c5-4f70-b09b-c72312711a86\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " Oct 31 01:23:11.679310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.679282 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca\") pod \"9aac2a03-80c5-4f70-b09b-c72312711a86\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " Oct 31 01:23:11.679531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.679391 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb\" (UniqueName: \"kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb\") pod \"9aac2a03-80c5-4f70-b09b-c72312711a86\" (UID: \"9aac2a03-80c5-4f70-b09b-c72312711a86\") " Oct 31 01:23:11.679638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.679613 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca" pod "9aac2a03-80c5-4f70-b09b-c72312711a86" (UID: "9aac2a03-80c5-4f70-b09b-c72312711a86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:23:11.679914 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.679893 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:23:11.681482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.681457 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca" pod "9aac2a03-80c5-4f70-b09b-c72312711a86" (UID: "9aac2a03-80c5-4f70-b09b-c72312711a86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:23:11.681587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.681523 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb" pod "9aac2a03-80c5-4f70-b09b-c72312711a86" (UID: "9aac2a03-80c5-4f70-b09b-c72312711a86"). InnerVolumeSpecName "kube-api-access-zzksb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:23:11.781475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.781442 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aac2a03-80c5-4f70-b09b-c72312711a86-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:23:11.781475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:11.781472 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzksb\" (UniqueName: \"kubernetes.io/projected/9aac2a03-80c5-4f70-b09b-c72312711a86-kube-api-access-zzksb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:23:12.428005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:12.427976 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" Oct 31 01:23:12.428167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:12.428002 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf" event={"ID":"9aac2a03-80c5-4f70-b09b-c72312711a86","Type":"ContainerDied","Data":"2e0d5fb42c22940eef6afcb1c50c9924d49a3a933a7f08812e225712e7b6e732"} Oct 31 01:23:12.428167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:12.428059 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0d5fb42c22940eef6afcb1c50c9924d49a3a933a7f08812e225712e7b6e732" Oct 31 01:23:30.612917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.612883 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9"] Oct 31 01:23:30.613679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.613658 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9aac2a03-80c5-4f70-b09b-c72312711a86" containerName="revoke-keys" Oct 31 01:23:30.613774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.613682 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aac2a03-80c5-4f70-b09b-c72312711a86" containerName="revoke-keys" Oct 31 01:23:30.613900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.613885 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aac2a03-80c5-4f70-b09b-c72312711a86" containerName="revoke-keys" Oct 31 01:23:30.617396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.617375 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.724781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.724750 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.724852 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzjm\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.724969 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.725008 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.725033 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725204 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.725178 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.725228 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.725374 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.725515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.725409 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.818168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.818135 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9"] Oct 31 01:23:30.827030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827001 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.827175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827092 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzjm\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.827175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827129 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.827175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827168 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.827391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827358 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827490 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827527 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827567 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827628 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827766 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827799 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.827825 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.828260 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.828805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.828660 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.830221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.830191 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.830357 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.830337 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.883383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.883313 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.883513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.883393 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzjm\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm\") pod \"e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:30.930754 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:30.930717 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:31.411761 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.411733 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9"] Oct 31 01:23:31.412333 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:23:31.412309 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85239cc5_b9d9_4237_b649_d27d998c9beb.slice/crio-802956e99c7ab5f2e0d6b64b424cd9adf09e3f2fdb1b482a84508345f0368be0 WatchSource:0}: Error finding container 802956e99c7ab5f2e0d6b64b424cd9adf09e3f2fdb1b482a84508345f0368be0: Status 404 returned error can't find the container with id 802956e99c7ab5f2e0d6b64b424cd9adf09e3f2fdb1b482a84508345f0368be0 Oct 31 01:23:31.415146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.415107 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:23:31.415226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.415195 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:23:31.415265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.415238 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:23:31.558674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.558631 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" event={"ID":"85239cc5-b9d9-4237-b649-d27d998c9beb","Type":"ContainerStarted","Data":"3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145"} Oct 31 01:23:31.558895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.558695 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" event={"ID":"85239cc5-b9d9-4237-b649-d27d998c9beb","Type":"ContainerStarted","Data":"802956e99c7ab5f2e0d6b64b424cd9adf09e3f2fdb1b482a84508345f0368be0"} Oct 31 01:23:31.680042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.679943 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podStartSLOduration=1.6799287760000001 podStartE2EDuration="1.679928776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:31.676769055 +0000 UTC m=+1583.483736660" watchObservedRunningTime="2026-10-31 01:23:31.679928776 +0000 UTC m=+1583.486896313" Oct 31 01:23:31.931208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:31.931121 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:32.082316 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:23:32.082280 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:23:32.953771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:32.953739 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 01:23:32.954215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:32.953814 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 01:23:33.933885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:33.932255 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 01:23:33.933885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:33.932337 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 01:23:34.950923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:34.941074 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 01:23:34.950923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:34.941135 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 01:23:35.957518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:35.957466 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 01:23:35.958194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:35.958141 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 01:23:36.938889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:36.933198 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 01:23:36.938889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:36.933262 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 01:23:37.931858 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:37.931819 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 31 01:23:37.932345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:37.931933 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 31 01:23:38.936348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:38.936315 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:38.936751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:38.936726 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:38.937538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:38.937517 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:23:46.643957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.643923 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd"] Oct 31 01:23:46.648424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.648400 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.700916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.700698 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f57e85-openshift-default-dockercfg-6bjhs\"" Oct 31 01:23:46.750161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750131 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750167 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750314 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750485 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750464 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750511 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750577 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjtc\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750629 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750729 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750711 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.750826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.750813 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.759434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.759358 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd"] Oct 31 01:23:46.851683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851650 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.851927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851771 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.851927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851791 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.851927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851825 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851937 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851965 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.851988 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjtc\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.852033 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.852116 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.852141 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.852200 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.852234 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.852556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.852527 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.853278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.853243 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.855455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.855413 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.855883 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.855841 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.910112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.910020 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjtc\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.910306 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.910283 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token\") pod \"e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:46.967149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:46.967118 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:47.411294 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:23:47.408485 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10abdb0d_7e36_4d79_bd39_4f82b09560df.slice/crio-8f2c52aa17f304a28f0cc152193a22b13cd5be3218c654b7f826f9a646fce31a WatchSource:0}: Error finding container 8f2c52aa17f304a28f0cc152193a22b13cd5be3218c654b7f826f9a646fce31a: Status 404 returned error can't find the container with id 8f2c52aa17f304a28f0cc152193a22b13cd5be3218c654b7f826f9a646fce31a Oct 31 01:23:47.411294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.408762 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd"] Oct 31 01:23:47.411751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.411709 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:23:47.411940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.411794 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:23:47.411940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.411837 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:23:47.677149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.677049 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" event={"ID":"10abdb0d-7e36-4d79-bd39-4f82b09560df","Type":"ContainerStarted","Data":"ac12fafacc6fc495c3a9c7cd4f04bb1f3940a55cd6d5195c329fe42c8f1c0786"} Oct 31 01:23:47.677149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.677111 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" event={"ID":"10abdb0d-7e36-4d79-bd39-4f82b09560df","Type":"ContainerStarted","Data":"8f2c52aa17f304a28f0cc152193a22b13cd5be3218c654b7f826f9a646fce31a"} Oct 31 01:23:47.793298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.793245 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podStartSLOduration=1.793231251 podStartE2EDuration="1.793231251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:23:47.790583304 +0000 UTC m=+1599.597550849" watchObservedRunningTime="2026-10-31 01:23:47.793231251 +0000 UTC m=+1599.600198793" Oct 31 01:23:47.967554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:47.967469 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:48.973149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:48.973120 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:49.692322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:49.692296 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:23:49.693167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:23:49.693145 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:24:02.066902 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:24:02.066848 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:24:05.976809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:05.976775 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:24:11.050036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:11.049995 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4"] Oct 31 01:24:11.098332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:11.098294 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979b5c9-pcqt4"] Oct 31 01:24:12.312601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:12.312566 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c143d7-3e4a-4b79-8080-58ed1909aab6" path="/var/lib/kubelet/pods/60c143d7-3e4a-4b79-8080-58ed1909aab6/volumes" Oct 31 01:24:19.614289 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.614204 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh"] Oct 31 01:24:19.621998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.621970 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.675696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.675633 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh"] Oct 31 01:24:19.720068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.720038 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:24:19.720226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.720093 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:24:19.752982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.752952 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.752982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.752983 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.753213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.753131 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jxd\" (UniqueName: \"kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.854246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.854217 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.854444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.854263 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.854444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.854362 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jxd\" (UniqueName: \"kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.855119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.855075 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.856794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.856767 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.911134 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.911066 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jxd\" (UniqueName: \"kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd\") pod \"maas-api-revoke-sub-b0a3ce49-rn8fh\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:19.936756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:19.936735 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:20.314669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:20.314638 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh"] Oct 31 01:24:20.314807 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:24:20.314697 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29740221_25b6_4367_857f_67a9703c933e.slice/crio-2a472895d0b661582b7f65589c7598fa315de41f9b5afee092ce6a932f460ab5 WatchSource:0}: Error finding container 2a472895d0b661582b7f65589c7598fa315de41f9b5afee092ce6a932f460ab5: Status 404 returned error can't find the container with id 2a472895d0b661582b7f65589c7598fa315de41f9b5afee092ce6a932f460ab5 Oct 31 01:24:21.002388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:21.002290 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" event={"ID":"29740221-25b6-4367-857f-67a9703c933e","Type":"ContainerStarted","Data":"f0943cc27a81221b8ae09cc60a148a4eefac75bbf6c082aeb9758a01ee741e4b"} Oct 31 01:24:21.002388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:21.002347 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" event={"ID":"29740221-25b6-4367-857f-67a9703c933e","Type":"ContainerStarted","Data":"2a472895d0b661582b7f65589c7598fa315de41f9b5afee092ce6a932f460ab5"} Oct 31 01:24:21.117019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:21.116966 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" podStartSLOduration=2.116951752 podStartE2EDuration="2.116951752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:21.115339132 +0000 UTC m=+1632.922306674" watchObservedRunningTime="2026-10-31 01:24:21.116951752 +0000 UTC m=+1632.923919298" Oct 31 01:24:22.011103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:22.011069 5756 generic.go:356] "Generic (PLEG): container finished" podID="29740221-25b6-4367-857f-67a9703c933e" containerID="f0943cc27a81221b8ae09cc60a148a4eefac75bbf6c082aeb9758a01ee741e4b" exitCode=0 Oct 31 01:24:22.011494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:22.011125 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" event={"ID":"29740221-25b6-4367-857f-67a9703c933e","Type":"ContainerDied","Data":"f0943cc27a81221b8ae09cc60a148a4eefac75bbf6c082aeb9758a01ee741e4b"} Oct 31 01:24:23.045654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.045620 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q"] Oct 31 01:24:23.096741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.096676 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4fa394b-s5z4q"] Oct 31 01:24:23.177022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.176993 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:23.299905 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.299826 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca\") pod \"29740221-25b6-4367-857f-67a9703c933e\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " Oct 31 01:24:23.300113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.299981 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd\" (UniqueName: \"kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd\") pod \"29740221-25b6-4367-857f-67a9703c933e\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " Oct 31 01:24:23.300197 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.300177 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca\") pod \"29740221-25b6-4367-857f-67a9703c933e\" (UID: \"29740221-25b6-4367-857f-67a9703c933e\") " Oct 31 01:24:23.301632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.301471 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca" pod "29740221-25b6-4367-857f-67a9703c933e" (UID: "29740221-25b6-4367-857f-67a9703c933e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:24:23.315749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.315713 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd" pod "29740221-25b6-4367-857f-67a9703c933e" (UID: "29740221-25b6-4367-857f-67a9703c933e"). InnerVolumeSpecName "kube-api-access-z5jxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:23.320424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.320395 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca" pod "29740221-25b6-4367-857f-67a9703c933e" (UID: "29740221-25b6-4367-857f-67a9703c933e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:24:23.403071 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.403040 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29740221-25b6-4367-857f-67a9703c933e-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:24:23.403071 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.403074 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29740221-25b6-4367-857f-67a9703c933e-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:24:23.403270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:23.403091 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5jxd\" (UniqueName: \"kubernetes.io/projected/29740221-25b6-4367-857f-67a9703c933e-kube-api-access-z5jxd\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:24:24.028664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:24.028635 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" Oct 31 01:24:24.028876 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:24.028666 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh" event={"ID":"29740221-25b6-4367-857f-67a9703c933e","Type":"ContainerDied","Data":"2a472895d0b661582b7f65589c7598fa315de41f9b5afee092ce6a932f460ab5"} Oct 31 01:24:24.028876 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:24.028750 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a472895d0b661582b7f65589c7598fa315de41f9b5afee092ce6a932f460ab5" Oct 31 01:24:24.312346 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:24.312313 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93516cb-00d8-45b0-bc3c-52808de0c24d" path="/var/lib/kubelet/pods/f93516cb-00d8-45b0-bc3c-52808de0c24d/volumes" Oct 31 01:24:27.047856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:27.047818 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b"] Oct 31 01:24:27.097172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:27.097138 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92aabbf6-7vq5b"] Oct 31 01:24:28.312427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:28.312395 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eda98d07-4f51-47a0-b44a-fa95ed4fc338" path="/var/lib/kubelet/pods/eda98d07-4f51-47a0-b44a-fa95ed4fc338/volumes" Oct 31 01:24:31.776453 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.776417 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98"] Oct 31 01:24:31.777354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.777331 5756 cpu_manager.go:414] "containerMap: removing container" podUID="29740221-25b6-4367-857f-67a9703c933e" containerName="revoke-keys" Oct 31 01:24:31.777418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.777359 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29740221-25b6-4367-857f-67a9703c933e" containerName="revoke-keys" Oct 31 01:24:31.777575 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.777563 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="29740221-25b6-4367-857f-67a9703c933e" containerName="revoke-keys" Oct 31 01:24:31.781972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.781946 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828467 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828522 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828645 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828699 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828820 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828936 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm45s\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.828985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.828973 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.829096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.829023 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.829156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.829116 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.834210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.834185 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-88d9b8-openshift-default-dockercfg-624v6\"" Oct 31 01:24:31.888618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.888582 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98"] Oct 31 01:24:31.930892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.930824 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.930905 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.930968 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.930999 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931094 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931171 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm45s\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931259 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931314 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931410 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931454 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931487 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.931535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.931507 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.932114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.932087 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.932247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.932228 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.933769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.933746 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.934165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.934144 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.988354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.988280 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:31.988354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:31.988346 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm45s\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s\") pod \"e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:32.081806 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:24:32.081731 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:24:32.098942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:32.098908 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:32.559764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:32.559739 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98"] Oct 31 01:24:32.561330 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:24:32.561306 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b281e1_9c63_443a_923f_86f06b782f29.slice/crio-f582f4a1fcc4a1a94297c2d7119eb1671180f6a5be06e5b875a74c49f60b4c4f WatchSource:0}: Error finding container f582f4a1fcc4a1a94297c2d7119eb1671180f6a5be06e5b875a74c49f60b4c4f: Status 404 returned error can't find the container with id f582f4a1fcc4a1a94297c2d7119eb1671180f6a5be06e5b875a74c49f60b4c4f Oct 31 01:24:32.564811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:32.564773 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:24:32.565025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:32.565001 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:24:32.565201 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:32.565174 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:24:33.047914 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:33.047887 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5"] Oct 31 01:24:33.051177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:33.051154 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff67da5f-mh8r5"] Oct 31 01:24:33.101388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:33.101352 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" event={"ID":"34b281e1-9c63-443a-923f-86f06b782f29","Type":"ContainerStarted","Data":"930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4"} Oct 31 01:24:33.101566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:33.101404 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" event={"ID":"34b281e1-9c63-443a-923f-86f06b782f29","Type":"ContainerStarted","Data":"f582f4a1fcc4a1a94297c2d7119eb1671180f6a5be06e5b875a74c49f60b4c4f"} Oct 31 01:24:33.221644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:33.221591 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podStartSLOduration=2.221578333 podStartE2EDuration="2.221578333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:33.218347498 +0000 UTC m=+1645.025315042" watchObservedRunningTime="2026-10-31 01:24:33.221578333 +0000 UTC m=+1645.028545876" Oct 31 01:24:34.099033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:34.098996 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:34.104855 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:34.104833 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:34.107984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:34.107964 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:34.108716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:34.108700 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:24:34.312360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:34.312329 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0d1420-7ad4-45e7-a174-71db7c246361" path="/var/lib/kubelet/pods/6d0d1420-7ad4-45e7-a174-71db7c246361/volumes" Oct 31 01:24:41.048288 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:41.047921 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc"] Oct 31 01:24:41.094930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:41.094898 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f72634-klrtc"] Oct 31 01:24:42.313535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:42.313501 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cef7c3e-0993-4d3d-9ead-be79531443ce" path="/var/lib/kubelet/pods/2cef7c3e-0993-4d3d-9ead-be79531443ce/volumes" Oct 31 01:24:47.050037 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:47.049996 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql"] Oct 31 01:24:47.097292 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:47.097261 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2623bb2e-bdqql"] Oct 31 01:24:47.389582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:47.389488 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:24:48.314152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:48.314112 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153ffb2f-d944-4be3-9a2d-6105864ac62b" path="/var/lib/kubelet/pods/153ffb2f-d944-4be3-9a2d-6105864ac62b/volumes" Oct 31 01:24:50.717994 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:50.717954 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:24:50.878889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:50.877972 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 01:24:52.064219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:52.064187 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj"] Oct 31 01:24:52.112541 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:52.112503 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f35019-8s4sj"] Oct 31 01:24:52.313123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:52.313092 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e593ea0-02ea-402c-ba1b-2249cc3c1a2f" path="/var/lib/kubelet/pods/5e593ea0-02ea-402c-ba1b-2249cc3c1a2f/volumes" Oct 31 01:24:59.368745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.368714 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7918-2eb58d"] Oct 31 01:24:59.371875 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.371838 5756 generic.go:356] "Generic (PLEG): container finished" podID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerID="ac12fafacc6fc495c3a9c7cd4f04bb1f3940a55cd6d5195c329fe42c8f1c0786" exitCode=0 Oct 31 01:24:59.372003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.371893 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" event={"ID":"10abdb0d-7e36-4d79-bd39-4f82b09560df","Type":"ContainerDied","Data":"ac12fafacc6fc495c3a9c7cd4f04bb1f3940a55cd6d5195c329fe42c8f1c0786"} Oct 31 01:24:59.372056 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.372030 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:24:59.372373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.372359 5756 scope.go:122] "RemoveContainer" containerID="ac12fafacc6fc495c3a9c7cd4f04bb1f3940a55cd6d5195c329fe42c8f1c0786" Oct 31 01:24:59.373975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.373948 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:24:59.374076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.374006 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:24:59.374076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.374036 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:24:59.427233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.427208 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-2eb58d"] Oct 31 01:24:59.473363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.473341 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vxhdm\"" Oct 31 01:24:59.475092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.475069 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6cwc\" (UniqueName: \"kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc\") pod \"test-curl-7918-2eb58d\" (UID: \"35a9b499-e83b-4835-92f1-4ba04f429f33\") " pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:24:59.577399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.577362 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6cwc\" (UniqueName: \"kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc\") pod \"test-curl-7918-2eb58d\" (UID: \"35a9b499-e83b-4835-92f1-4ba04f429f33\") " pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:24:59.639609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.639530 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6cwc\" (UniqueName: \"kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc\") pod \"test-curl-7918-2eb58d\" (UID: \"35a9b499-e83b-4835-92f1-4ba04f429f33\") " pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:24:59.684566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:24:59.684534 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:00.067022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.066990 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-2eb58d"] Oct 31 01:25:00.067733 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:25:00.067698 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35a9b499_e83b_4835_92f1_4ba04f429f33.slice/crio-ede018c0d9ec99a869e51f046a649d9a2e50230639add40035e9ac3851f8d42b WatchSource:0}: Error finding container ede018c0d9ec99a869e51f046a649d9a2e50230639add40035e9ac3851f8d42b: Status 404 returned error can't find the container with id ede018c0d9ec99a869e51f046a649d9a2e50230639add40035e9ac3851f8d42b Oct 31 01:25:00.069849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.069830 5756 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:25:00.379982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.379889 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-2eb58d" event={"ID":"35a9b499-e83b-4835-92f1-4ba04f429f33","Type":"ContainerStarted","Data":"ede018c0d9ec99a869e51f046a649d9a2e50230639add40035e9ac3851f8d42b"} Oct 31 01:25:00.382011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.381984 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" event={"ID":"10abdb0d-7e36-4d79-bd39-4f82b09560df","Type":"ContainerStarted","Data":"7f95f303608648880bffc55e68d30c2c0d6f807617b51d9e8f15754d5cf623fd"} Oct 31 01:25:00.967610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.967587 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:25:00.968696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.968672 5756 patch_prober.go:29] interesting pod/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 01:25:00.968808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:00.968732 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 01:25:01.390394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:01.390358 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-2eb58d" event={"ID":"35a9b499-e83b-4835-92f1-4ba04f429f33","Type":"ContainerStarted","Data":"b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10"} Oct 31 01:25:01.510223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:01.510167 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7918-2eb58d" podStartSLOduration=1.672929401 podStartE2EDuration="2.51015239s" totalImagesPullingTime="837.222989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:01.508351366 +0000 UTC m=+1673.315318911" watchObservedRunningTime="2026-10-31 01:25:01.51015239 +0000 UTC m=+1673.317119933" Oct 31 01:25:01.967441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:01.967405 5756 patch_prober.go:29] interesting pod/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 01:25:01.967631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:01.967454 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 01:25:02.078820 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:02.078781 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:25:02.968348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:02.968313 5756 patch_prober.go:29] interesting pod/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 01:25:02.968749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:02.968366 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 01:25:03.968053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:03.968006 5756 patch_prober.go:29] interesting pod/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 01:25:03.968398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:03.968366 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 01:25:04.697484 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:04.697454 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:25:04.967699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:04.967608 5756 patch_prober.go:29] interesting pod/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 01:25:04.967699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:04.967678 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 01:25:05.973921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:05.973857 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:25:05.975006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:05.974987 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:25:08.808203 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:08.808169 5756 scope.go:122] "RemoveContainer" containerID="2841a5956c41eb49b326dcfff4d59e7270e624d6f23e466bffea48cc52fe089e" Oct 31 01:25:10.313660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.313615 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r"] Oct 31 01:25:10.319982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.319956 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.363826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.363722 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpkf\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.364213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.364188 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.364450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.364434 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.364912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.364890 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.365338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.365313 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.365549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.365529 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.365713 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.365689 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.366073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.366054 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.375058 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.375028 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.383107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.383080 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-02c8fd-openshift-default-dockercfg-n7v4x\"" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487524 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487594 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487710 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpkf\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487750 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487798 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487847 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487922 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.487984 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.488427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.488019 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.514804 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.514771 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.516433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.515447 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.516590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.516211 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.519303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.519274 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.527705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.520821 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.528005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.527847 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.531977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.528225 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.535322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.535280 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r"] Oct 31 01:25:10.573840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.573773 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpkf\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.575838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.575810 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token\") pod \"e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:10.662184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:10.662151 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:11.090405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.090371 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s"] Oct 31 01:25:11.094649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.094616 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98a3be0e-7gn8s"] Oct 31 01:25:11.125608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.125580 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7918-2eb58d"] Oct 31 01:25:11.125801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.125774 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7918-2eb58d" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" containerName="test-curl-7918-2eb58d" containerID="cri-o://b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10" gracePeriod=2 Oct 31 01:25:11.129682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.129654 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7918-2eb58d"] Oct 31 01:25:11.170433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.170400 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r"] Oct 31 01:25:11.176307 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:11.176276 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-2eb58d\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:11.177356 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:25:11.177328 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c06d50_b4bf_4682_b178_ed2886ed5f85.slice/crio-7aa9e62126492d642045aecc34d2d293753fbe5de5ef3fa5c718a0db6618f33e WatchSource:0}: Error finding container 7aa9e62126492d642045aecc34d2d293753fbe5de5ef3fa5c718a0db6618f33e: Status 404 returned error can't find the container with id 7aa9e62126492d642045aecc34d2d293753fbe5de5ef3fa5c718a0db6618f33e Oct 31 01:25:11.180984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.180945 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:25:11.181061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.181035 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:25:11.181100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.181079 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:25:11.532639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.532600 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" event={"ID":"c2c06d50-b4bf-4682-b178-ed2886ed5f85","Type":"ContainerStarted","Data":"452b80215b19bc3ccce3a4599ef6253a56073be7b20708bc2a3f3f17b4398a15"} Oct 31 01:25:11.533274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.532669 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" event={"ID":"c2c06d50-b4bf-4682-b178-ed2886ed5f85","Type":"ContainerStarted","Data":"7aa9e62126492d642045aecc34d2d293753fbe5de5ef3fa5c718a0db6618f33e"} Oct 31 01:25:11.583530 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:11.583485 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-2eb58d\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:11.662522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.662492 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:11.701470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:11.701411 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podStartSLOduration=1.701391981 podStartE2EDuration="1.701391981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:11.700210985 +0000 UTC m=+1683.507178529" watchObservedRunningTime="2026-10-31 01:25:11.701391981 +0000 UTC m=+1683.508359535" Oct 31 01:25:12.074906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.074827 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h"] Oct 31 01:25:12.075531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.075509 5756 cpu_manager.go:414] "containerMap: removing container" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" containerName="test-curl-7918-2eb58d" Oct 31 01:25:12.075634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.075534 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" containerName="test-curl-7918-2eb58d" Oct 31 01:25:12.075728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.075714 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" containerName="test-curl-7918-2eb58d" Oct 31 01:25:12.079251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.079229 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.113050 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.113020 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.113166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.113076 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.113292 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.113272 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgfz\" (UniqueName: \"kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.135105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.135076 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:25:12.135259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.135132 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:25:12.138245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.138220 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h"] Oct 31 01:25:12.214807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.214773 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.214996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.214854 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.215081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.215045 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgfz\" (UniqueName: \"kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.215544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.215519 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.217434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.217414 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.248045 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:12.248006 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-2eb58d\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:12.272922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.272894 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgfz\" (UniqueName: \"kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-r468h\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.312740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.312710 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18a41c7-983a-49b2-81d1-699f9fbcc4f3" path="/var/lib/kubelet/pods/c18a41c7-983a-49b2-81d1-699f9fbcc4f3/volumes" Oct 31 01:25:12.394003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.393905 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:12.667915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.667832 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:12.777939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:12.777918 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h"] Oct 31 01:25:12.779432 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:25:12.779405 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9546bc9c_e504_4985_ad38_f4a327ceb9eb.slice/crio-b64250cc3be71f399dbc0576b1326925cc462eadefae02bbf84f4367c37c5790 WatchSource:0}: Error finding container b64250cc3be71f399dbc0576b1326925cc462eadefae02bbf84f4367c37c5790: Status 404 returned error can't find the container with id b64250cc3be71f399dbc0576b1326925cc462eadefae02bbf84f4367c37c5790 Oct 31 01:25:13.372406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.372381 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:13.432344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.432319 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc\" (UniqueName: \"kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc\") pod \"35a9b499-e83b-4835-92f1-4ba04f429f33\" (UID: \"35a9b499-e83b-4835-92f1-4ba04f429f33\") " Oct 31 01:25:13.434482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.434455 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc" pod "35a9b499-e83b-4835-92f1-4ba04f429f33" (UID: "35a9b499-e83b-4835-92f1-4ba04f429f33"). InnerVolumeSpecName "kube-api-access-n6cwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:13.534225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.534196 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6cwc\" (UniqueName: \"kubernetes.io/projected/35a9b499-e83b-4835-92f1-4ba04f429f33-kube-api-access-n6cwc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:13.548360 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.548326 5756 generic.go:356] "Generic (PLEG): container finished" podID="35a9b499-e83b-4835-92f1-4ba04f429f33" containerID="b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10" exitCode=137 Oct 31 01:25:13.548532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.548371 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:13.548532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.548423 5756 scope.go:122] "RemoveContainer" containerID="b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10" Oct 31 01:25:13.550234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.550189 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" event={"ID":"9546bc9c-e504-4985-ad38-f4a327ceb9eb","Type":"ContainerStarted","Data":"a0e5038862404d0577438e99884744b496241ba52d8c6222072aba38fbdcb16a"} Oct 31 01:25:13.550358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.550248 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" event={"ID":"9546bc9c-e504-4985-ad38-f4a327ceb9eb","Type":"ContainerStarted","Data":"b64250cc3be71f399dbc0576b1326925cc462eadefae02bbf84f4367c37c5790"} Oct 31 01:25:13.550358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.550298 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:13.551594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.551575 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:25:13.559224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.559210 5756 scope.go:122] "RemoveContainer" containerID="b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10" Oct 31 01:25:13.559503 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:13.559482 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10\": container with ID starting with b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10 not found: ID does not exist" containerID="b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10" Oct 31 01:25:13.559572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.559512 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10"} err="failed to get container status \"b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10\": rpc error: code = NotFound desc = could not find container \"b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10\": container with ID starting with b4c5ed967c29b4c6394435192f1643d3f12ecda852b71bfbd2aee15ca64e5e10 not found: ID does not exist" Oct 31 01:25:13.666364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:13.666316 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" podStartSLOduration=1.666302918 podStartE2EDuration="1.666302918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:13.664179258 +0000 UTC m=+1685.471146802" watchObservedRunningTime="2026-10-31 01:25:13.666302918 +0000 UTC m=+1685.473270518" Oct 31 01:25:13.715849 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:13.715810 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-2eb58d\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" pod="opendatahub/test-curl-7918-2eb58d" Oct 31 01:25:14.312433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:14.312402 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a9b499-e83b-4835-92f1-4ba04f429f33" path="/var/lib/kubelet/pods/35a9b499-e83b-4835-92f1-4ba04f429f33/volumes" Oct 31 01:25:14.559297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:14.559264 5756 generic.go:356] "Generic (PLEG): container finished" podID="9546bc9c-e504-4985-ad38-f4a327ceb9eb" containerID="a0e5038862404d0577438e99884744b496241ba52d8c6222072aba38fbdcb16a" exitCode=0 Oct 31 01:25:14.559467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:14.559327 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" event={"ID":"9546bc9c-e504-4985-ad38-f4a327ceb9eb","Type":"ContainerDied","Data":"a0e5038862404d0577438e99884744b496241ba52d8c6222072aba38fbdcb16a"} Oct 31 01:25:15.729383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.729357 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:15.766746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.766715 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca\") pod \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " Oct 31 01:25:15.766942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.766840 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz\" (UniqueName: \"kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz\") pod \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " Oct 31 01:25:15.766942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.766913 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca\") pod \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\" (UID: \"9546bc9c-e504-4985-ad38-f4a327ceb9eb\") " Oct 31 01:25:15.767350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.767320 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca" pod "9546bc9c-e504-4985-ad38-f4a327ceb9eb" (UID: "9546bc9c-e504-4985-ad38-f4a327ceb9eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:15.767614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.767592 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:15.769169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.769147 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz" pod "9546bc9c-e504-4985-ad38-f4a327ceb9eb" (UID: "9546bc9c-e504-4985-ad38-f4a327ceb9eb"). InnerVolumeSpecName "kube-api-access-lkgfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:15.769245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.769181 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca" pod "9546bc9c-e504-4985-ad38-f4a327ceb9eb" (UID: "9546bc9c-e504-4985-ad38-f4a327ceb9eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:15.868689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.868617 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgfz\" (UniqueName: \"kubernetes.io/projected/9546bc9c-e504-4985-ad38-f4a327ceb9eb-kube-api-access-lkgfz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:15.868689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:15.868645 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9546bc9c-e504-4985-ad38-f4a327ceb9eb-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:16.576643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:16.576565 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" Oct 31 01:25:16.576643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:16.576592 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h" event={"ID":"9546bc9c-e504-4985-ad38-f4a327ceb9eb","Type":"ContainerDied","Data":"b64250cc3be71f399dbc0576b1326925cc462eadefae02bbf84f4367c37c5790"} Oct 31 01:25:16.576643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:16.576636 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b64250cc3be71f399dbc0576b1326925cc462eadefae02bbf84f4367c37c5790" Oct 31 01:25:17.057698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:17.057655 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z"] Oct 31 01:25:17.106823 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:17.106795 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53340def-6fl5z"] Oct 31 01:25:18.320681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:18.320637 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8650adf2-94a1-43d1-815e-1c525cd845ca" path="/var/lib/kubelet/pods/8650adf2-94a1-43d1-815e-1c525cd845ca/volumes" Oct 31 01:25:21.102230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:21.102195 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4"] Oct 31 01:25:21.151553 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:21.151519 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea9a848b-hbbx4"] Oct 31 01:25:22.313213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:22.313181 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec4c76a-2537-4252-879e-610d34abcff5" path="/var/lib/kubelet/pods/2ec4c76a-2537-4252-879e-610d34abcff5/volumes" Oct 31 01:25:25.458185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:25.458153 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:25:32.067726 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:25:32.067681 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:25:34.056745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:34.056710 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln"] Oct 31 01:25:34.106118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:34.106088 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b65a9483-45kln"] Oct 31 01:25:34.320419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:34.320343 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55808545-23d5-4d0e-8d7f-f1364516c51b" path="/var/lib/kubelet/pods/55808545-23d5-4d0e-8d7f-f1364516c51b/volumes" Oct 31 01:25:38.695355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.695320 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6"] Oct 31 01:25:38.696017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.696001 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9546bc9c-e504-4985-ad38-f4a327ceb9eb" containerName="revoke-keys" Oct 31 01:25:38.696017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.696017 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9546bc9c-e504-4985-ad38-f4a327ceb9eb" containerName="revoke-keys" Oct 31 01:25:38.696182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.696172 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9546bc9c-e504-4985-ad38-f4a327ceb9eb" containerName="revoke-keys" Oct 31 01:25:38.700616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.700593 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.770895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.768204 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-987f2f-openshift-default-dockercfg-q6wsb\"" Oct 31 01:25:38.823675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.823630 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6"] Oct 31 01:25:38.833677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.833634 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.833970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.833944 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834003 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834067 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56l4n\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834193 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834246 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834542 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834607 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.834711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.834695 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.936317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.936286 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937017 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937265 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937424 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937546 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937673 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937788 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56l4n\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.937985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.937966 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.938122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.938106 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.938265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.938247 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.938358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.938011 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.938441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.936952 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.938801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.938776 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.939917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.939117 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.949190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.945384 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.949190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.947092 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.996397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.996359 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56l4n\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:38.999190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:38.999160 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token\") pod \"e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:39.017090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.017058 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:39.475713 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.475679 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6"] Oct 31 01:25:39.476573 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:25:39.476534 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1200043_f2f0_45c1_8620_ac65307b0bdc.slice/crio-145c19599f9341135deff3005697ff7c0c97037423c140dc34a991dc1b62dd57 WatchSource:0}: Error finding container 145c19599f9341135deff3005697ff7c0c97037423c140dc34a991dc1b62dd57: Status 404 returned error can't find the container with id 145c19599f9341135deff3005697ff7c0c97037423c140dc34a991dc1b62dd57 Oct 31 01:25:39.480210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.480167 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:25:39.480304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.480265 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:25:39.480342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.480310 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:25:39.757214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.757116 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" event={"ID":"c1200043-f2f0-45c1-8620-ac65307b0bdc","Type":"ContainerStarted","Data":"f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86"} Oct 31 01:25:39.757214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.757191 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" event={"ID":"c1200043-f2f0-45c1-8620-ac65307b0bdc","Type":"ContainerStarted","Data":"145c19599f9341135deff3005697ff7c0c97037423c140dc34a991dc1b62dd57"} Oct 31 01:25:39.875512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:39.875456 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podStartSLOduration=1.8754416470000002 podStartE2EDuration="1.875441647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:39.873530699 +0000 UTC m=+1711.680498245" watchObservedRunningTime="2026-10-31 01:25:39.875441647 +0000 UTC m=+1711.682409190" Oct 31 01:25:40.017296 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:40.017199 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:40.024719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:40.024689 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:40.765611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:40.765579 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:40.766666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:40.766646 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:25:47.290889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.290785 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588"] Oct 31 01:25:47.295569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.295544 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.349110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.349026 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:25:47.358418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.358383 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.358590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.358546 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9x57\" (UniqueName: \"kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.358655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.358600 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.398259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.398214 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:25:47.402781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.402739 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588"] Oct 31 01:25:47.459840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.459803 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.460038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.459932 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9x57\" (UniqueName: \"kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.460038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.459968 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.460494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.460460 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.462929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.462907 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.517514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.517479 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9x57\" (UniqueName: \"kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57\") pod \"maas-api-revoke-sub-37914ab1-zh588\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.611442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.611348 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:47.993541 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:47.993510 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588"] Oct 31 01:25:47.994112 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:25:47.994081 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f242e15_32d2_4f0e_b93f_f77993db4169.slice/crio-c6d2502817812e22f7c35116161aace7295c02e05c5261d5c54fb84e8ec4bd04 WatchSource:0}: Error finding container c6d2502817812e22f7c35116161aace7295c02e05c5261d5c54fb84e8ec4bd04: Status 404 returned error can't find the container with id c6d2502817812e22f7c35116161aace7295c02e05c5261d5c54fb84e8ec4bd04 Oct 31 01:25:48.836277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:48.836238 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" event={"ID":"6f242e15-32d2-4f0e-b93f-f77993db4169","Type":"ContainerStarted","Data":"d2c6822b1930616642d539cf0f58fb48e232a513a06683541582ae7f8f7db849"} Oct 31 01:25:48.836772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:48.836307 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" event={"ID":"6f242e15-32d2-4f0e-b93f-f77993db4169","Type":"ContainerStarted","Data":"c6d2502817812e22f7c35116161aace7295c02e05c5261d5c54fb84e8ec4bd04"} Oct 31 01:25:48.952142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:48.952088 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" podStartSLOduration=1.9520737769999998 podStartE2EDuration="1.952073777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:48.950104508 +0000 UTC m=+1720.757072052" watchObservedRunningTime="2026-10-31 01:25:48.952073777 +0000 UTC m=+1720.759041324" Oct 31 01:25:49.123028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.122951 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq"] Oct 31 01:25:49.127580 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.127558 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.182389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.182358 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq"] Oct 31 01:25:49.188955 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.188927 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgph\" (UniqueName: \"kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.189086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.189058 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.189334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.189315 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.291133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.291103 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.291320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.291163 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgph\" (UniqueName: \"kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.291320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.291215 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.291753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.291731 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.293797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.293772 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.348728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.348697 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgph\" (UniqueName: \"kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph\") pod \"maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.505136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.505055 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:49.845590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.845555 5756 generic.go:356] "Generic (PLEG): container finished" podID="6f242e15-32d2-4f0e-b93f-f77993db4169" containerID="d2c6822b1930616642d539cf0f58fb48e232a513a06683541582ae7f8f7db849" exitCode=0 Oct 31 01:25:49.846024 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.845632 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" event={"ID":"6f242e15-32d2-4f0e-b93f-f77993db4169","Type":"ContainerDied","Data":"d2c6822b1930616642d539cf0f58fb48e232a513a06683541582ae7f8f7db849"} Oct 31 01:25:49.884504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:49.884477 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq"] Oct 31 01:25:49.885305 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:25:49.885281 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9697036c_a4d8_425f_ac2d_837e3cd29400.slice/crio-d4c67e32674e4d6374cea1ddb497671f34751c23f02a4f32d8dc2bd9deca6604 WatchSource:0}: Error finding container d4c67e32674e4d6374cea1ddb497671f34751c23f02a4f32d8dc2bd9deca6604: Status 404 returned error can't find the container with id d4c67e32674e4d6374cea1ddb497671f34751c23f02a4f32d8dc2bd9deca6604 Oct 31 01:25:50.856932 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:50.856898 5756 generic.go:356] "Generic (PLEG): container finished" podID="9697036c-a4d8-425f-ac2d-837e3cd29400" containerID="d9cc72e8d3bd91ea2f86c7c08ed52a7188a42c3388c88dc8f0fce50646dd6b96" exitCode=0 Oct 31 01:25:50.857257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:50.856982 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" event={"ID":"9697036c-a4d8-425f-ac2d-837e3cd29400","Type":"ContainerDied","Data":"d9cc72e8d3bd91ea2f86c7c08ed52a7188a42c3388c88dc8f0fce50646dd6b96"} Oct 31 01:25:50.857257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:50.857035 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" event={"ID":"9697036c-a4d8-425f-ac2d-837e3cd29400","Type":"ContainerStarted","Data":"d4c67e32674e4d6374cea1ddb497671f34751c23f02a4f32d8dc2bd9deca6604"} Oct 31 01:25:51.047157 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.047128 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:51.050194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.050167 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88"] Oct 31 01:25:51.053332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.053312 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6380bd2-fzc88"] Oct 31 01:25:51.117887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.117794 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57\" (UniqueName: \"kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57\") pod \"6f242e15-32d2-4f0e-b93f-f77993db4169\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " Oct 31 01:25:51.118019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.117948 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca\") pod \"6f242e15-32d2-4f0e-b93f-f77993db4169\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " Oct 31 01:25:51.118097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.118080 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca\") pod \"6f242e15-32d2-4f0e-b93f-f77993db4169\" (UID: \"6f242e15-32d2-4f0e-b93f-f77993db4169\") " Oct 31 01:25:51.118418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.118392 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca" pod "6f242e15-32d2-4f0e-b93f-f77993db4169" (UID: "6f242e15-32d2-4f0e-b93f-f77993db4169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:51.118687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.118665 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:51.120549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.120524 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca" pod "6f242e15-32d2-4f0e-b93f-f77993db4169" (UID: "6f242e15-32d2-4f0e-b93f-f77993db4169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:51.120636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.120547 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57" pod "6f242e15-32d2-4f0e-b93f-f77993db4169" (UID: "6f242e15-32d2-4f0e-b93f-f77993db4169"). InnerVolumeSpecName "kube-api-access-z9x57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:51.219775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.219743 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9x57\" (UniqueName: \"kubernetes.io/projected/6f242e15-32d2-4f0e-b93f-f77993db4169-kube-api-access-z9x57\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:51.219775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.219775 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f242e15-32d2-4f0e-b93f-f77993db4169-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:51.866995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.866967 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" Oct 31 01:25:51.866995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.866971 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588" event={"ID":"6f242e15-32d2-4f0e-b93f-f77993db4169","Type":"ContainerDied","Data":"c6d2502817812e22f7c35116161aace7295c02e05c5261d5c54fb84e8ec4bd04"} Oct 31 01:25:51.867528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:51.867028 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d2502817812e22f7c35116161aace7295c02e05c5261d5c54fb84e8ec4bd04" Oct 31 01:25:52.019301 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.019274 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:52.159939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.154419 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca\") pod \"9697036c-a4d8-425f-ac2d-837e3cd29400\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " Oct 31 01:25:52.159939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.154675 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph\" (UniqueName: \"kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph\") pod \"9697036c-a4d8-425f-ac2d-837e3cd29400\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " Oct 31 01:25:52.159939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.154712 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca\") pod \"9697036c-a4d8-425f-ac2d-837e3cd29400\" (UID: \"9697036c-a4d8-425f-ac2d-837e3cd29400\") " Oct 31 01:25:52.167481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.167435 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca" pod "9697036c-a4d8-425f-ac2d-837e3cd29400" (UID: "9697036c-a4d8-425f-ac2d-837e3cd29400"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:25:52.196844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.196805 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca" pod "9697036c-a4d8-425f-ac2d-837e3cd29400" (UID: "9697036c-a4d8-425f-ac2d-837e3cd29400"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:25:52.198382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.198339 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph" pod "9697036c-a4d8-425f-ac2d-837e3cd29400" (UID: "9697036c-a4d8-425f-ac2d-837e3cd29400"). InnerVolumeSpecName "kube-api-access-zvgph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:25:52.257061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.257025 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:52.257061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.257062 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvgph\" (UniqueName: \"kubernetes.io/projected/9697036c-a4d8-425f-ac2d-837e3cd29400-kube-api-access-zvgph\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:52.257263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.257079 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9697036c-a4d8-425f-ac2d-837e3cd29400-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:25:52.315265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.315232 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713c3abe-806d-4293-8f3a-99c848ddc097" path="/var/lib/kubelet/pods/713c3abe-806d-4293-8f3a-99c848ddc097/volumes" Oct 31 01:25:52.876159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.876125 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" Oct 31 01:25:52.876159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.876148 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq" event={"ID":"9697036c-a4d8-425f-ac2d-837e3cd29400","Type":"ContainerDied","Data":"d4c67e32674e4d6374cea1ddb497671f34751c23f02a4f32d8dc2bd9deca6604"} Oct 31 01:25:52.876580 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:52.876193 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4c67e32674e4d6374cea1ddb497671f34751c23f02a4f32d8dc2bd9deca6604" Oct 31 01:25:58.059159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:58.059126 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s"] Oct 31 01:25:58.105133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:58.105101 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f829c1e-7qn6s"] Oct 31 01:25:58.312423 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:25:58.312331 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dcc976d-41bb-4a4e-b5c5-3d78712c511f" path="/var/lib/kubelet/pods/1dcc976d-41bb-4a4e-b5c5-3d78712c511f/volumes" Oct 31 01:26:02.111504 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:26:02.111438 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:26:03.965997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:03.965965 5756 generic.go:356] "Generic (PLEG): container finished" podID="34b281e1-9c63-443a-923f-86f06b782f29" containerID="930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4" exitCode=0 Oct 31 01:26:03.966550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:03.966048 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" event={"ID":"34b281e1-9c63-443a-923f-86f06b782f29","Type":"ContainerDied","Data":"930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4"} Oct 31 01:26:03.966642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:03.966618 5756 scope.go:122] "RemoveContainer" containerID="930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4" Oct 31 01:26:03.969793 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:03.969759 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:26:03.969926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:03.969843 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:26:03.969926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:03.969916 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:26:04.108544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:04.108510 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:26:04.975361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:04.975322 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" event={"ID":"34b281e1-9c63-443a-923f-86f06b782f29","Type":"ContainerStarted","Data":"c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57"} Oct 31 01:26:05.100097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:05.100059 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:26:05.101283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:05.101256 5756 patch_prober.go:29] interesting pod/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 01:26:05.101425 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:05.101314 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 01:26:06.108372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:06.108337 5756 patch_prober.go:29] interesting pod/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 01:26:06.109092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:06.108408 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 01:26:07.099419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:07.099385 5756 patch_prober.go:29] interesting pod/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 01:26:07.099605 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:07.099436 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 01:26:08.099843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.099812 5756 patch_prober.go:29] interesting pod/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 01:26:08.100276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.099900 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 01:26:08.852901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.850075 5756 scope.go:122] "RemoveContainer" containerID="9c9ef3baed98ae28065460237ac8706ab1559ef8ff652207a94fbb045229f517" Oct 31 01:26:08.866922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.866848 5756 scope.go:122] "RemoveContainer" containerID="be1a0568d274dbc14322580496a2e5d660158c774e9f162991e39123d21f3bad" Oct 31 01:26:08.876327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.876306 5756 scope.go:122] "RemoveContainer" containerID="3f3366be1094a428ba3d410cd477cfcc5cff3b4468d8c3325db553b34cd29238" Oct 31 01:26:08.885285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.885262 5756 scope.go:122] "RemoveContainer" containerID="fd7b4377ab9dca58f97dd6a56a19277e8caa4522995d07991f4f3e469263f6ba" Oct 31 01:26:08.894454 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.894436 5756 scope.go:122] "RemoveContainer" containerID="f8c41d47989db0608ac8301ff91c53d0a8f7fac662ba3e7632d879f4fd6d6ef2" Oct 31 01:26:08.902808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.902784 5756 scope.go:122] "RemoveContainer" containerID="37fcfda1f6445646641ed5f73a7c116d9ce488f43efa5ca6fb7215f2aae5f26d" Oct 31 01:26:08.911386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:08.911363 5756 scope.go:122] "RemoveContainer" containerID="b3a4ae29d03f7fb7d28802c7a34facf852defc852b5a9de442c8ab2088e58e9e" Oct 31 01:26:09.055140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:09.055106 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf"] Oct 31 01:26:09.099973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:09.099944 5756 patch_prober.go:29] interesting pod/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 31 01:26:09.100351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:09.099995 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 31 01:26:09.100957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:09.100938 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe762391-trxqf"] Oct 31 01:26:10.162273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:10.162240 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:26:10.162771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:10.162544 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:26:10.167600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:10.167556 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:26:10.312351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:10.312322 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c6b9b8-5c31-499b-a2ec-69b8a6e421f1" path="/var/lib/kubelet/pods/71c6b9b8-5c31-499b-a2ec-69b8a6e421f1/volumes" Oct 31 01:26:15.740447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:15.740413 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r"] Oct 31 01:26:15.740957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:15.740649 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" containerID="cri-o://dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a" gracePeriod=30 Oct 31 01:26:18.034240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:18.034203 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:26:20.485952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:20.485915 5756 patch_prober.go:29] interesting pod/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:26:20.486437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:20.485971 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:26:21.082434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.082401 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:26:21.093197 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.093165 5756 generic.go:356] "Generic (PLEG): container finished" podID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerID="dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a" exitCode=0 Oct 31 01:26:21.093329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.093210 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" Oct 31 01:26:21.093329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.093250 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" event={"ID":"3955cd72-831e-4360-a2b7-bef5dd3f36ea","Type":"ContainerDied","Data":"dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a"} Oct 31 01:26:21.093407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.093329 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r" event={"ID":"3955cd72-831e-4360-a2b7-bef5dd3f36ea","Type":"ContainerDied","Data":"168e6ef640c6ff8fcc100a72b526f0ac880243cb280b77107e47c99a0729b190"} Oct 31 01:26:21.093407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.093357 5756 scope.go:122] "RemoveContainer" containerID="dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a" Oct 31 01:26:21.107214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.107198 5756 scope.go:122] "RemoveContainer" containerID="dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a" Oct 31 01:26:21.107522 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:26:21.107502 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a\": container with ID starting with dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a not found: ID does not exist" containerID="dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a" Oct 31 01:26:21.107590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.107534 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a"} err="failed to get container status \"dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a\": rpc error: code = NotFound desc = could not find container \"dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a\": container with ID starting with dbf9ae3fb01d20d3bcf162a162641056e8b0662b89ee0d0d70f05b3368a1f86a not found: ID does not exist" Oct 31 01:26:21.173628 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173597 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.173811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173733 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.173811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173778 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173877 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173855 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:21.174031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173939 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.173987 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174050 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174076 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:21.174250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174195 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174214 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:21.174451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174251 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token\") pod \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\" (UID: \"3955cd72-831e-4360-a2b7-bef5dd3f36ea\") " Oct 31 01:26:21.174451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174265 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:21.174451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174340 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:21.174820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174797 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.175072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174825 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.175072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174904 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.175072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174923 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.175072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.174979 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.176674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.176644 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:21.176813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.176669 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:26:21.176931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.176726 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:21.176997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.176746 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg" pod "3955cd72-831e-4360-a2b7-bef5dd3f36ea" (UID: "3955cd72-831e-4360-a2b7-bef5dd3f36ea"). InnerVolumeSpecName "kube-api-access-m8zvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:21.276443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.276351 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zvg\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-kube-api-access-m8zvg\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.276443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.276386 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.276443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.276400 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.276443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.276413 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3955cd72-831e-4360-a2b7-bef5dd3f36ea-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:21.519147 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.519100 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r"] Oct 31 01:26:21.524544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:21.524518 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-4pc9r"] Oct 31 01:26:22.313221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:22.313181 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" path="/var/lib/kubelet/pods/3955cd72-831e-4360-a2b7-bef5dd3f36ea/volumes" Oct 31 01:26:28.551745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:28.551710 5756 patch_prober.go:29] interesting pod/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 01:26:28.552148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:28.551781 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 01:26:29.155659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:29.155625 5756 generic.go:356] "Generic (PLEG): container finished" podID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerID="452b80215b19bc3ccce3a4599ef6253a56073be7b20708bc2a3f3f17b4398a15" exitCode=0 Oct 31 01:26:29.155836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:29.155693 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" event={"ID":"c2c06d50-b4bf-4682-b178-ed2886ed5f85","Type":"ContainerDied","Data":"452b80215b19bc3ccce3a4599ef6253a56073be7b20708bc2a3f3f17b4398a15"} Oct 31 01:26:29.156136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:29.156121 5756 scope.go:122] "RemoveContainer" containerID="452b80215b19bc3ccce3a4599ef6253a56073be7b20708bc2a3f3f17b4398a15" Oct 31 01:26:29.157563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:29.157536 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:26:29.157658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:29.157602 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:26:29.157658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:29.157634 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:26:30.180607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:30.180563 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" event={"ID":"c2c06d50-b4bf-4682-b178-ed2886ed5f85","Type":"ContainerStarted","Data":"8fec5a7060c7d42fda89c7faf169fd26fb0afc676df124d21780f99f01ec05f9"} Oct 31 01:26:30.663211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:30.663168 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:26:30.663425 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:30.663388 5756 patch_prober.go:29] interesting pod/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 01:26:30.663494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:30.663440 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 01:26:31.056619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:31.056569 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2"] Oct 31 01:26:31.106385 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:31.106353 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466a8c0c-vsnc2"] Oct 31 01:26:31.663963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:31.663842 5756 patch_prober.go:29] interesting pod/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 01:26:31.663963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:31.663941 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 01:26:32.080144 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:26:32.080107 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:26:32.312450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:32.312417 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d060506-e764-4151-a9f2-220e058e7f4c" path="/var/lib/kubelet/pods/1d060506-e764-4151-a9f2-220e058e7f4c/volumes" Oct 31 01:26:32.662749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:32.662713 5756 patch_prober.go:29] interesting pod/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 01:26:32.662957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:32.662775 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 01:26:33.663237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:33.663200 5756 patch_prober.go:29] interesting pod/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 01:26:33.663631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:33.663256 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 01:26:34.663938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:34.663896 5756 patch_prober.go:29] interesting pod/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 01:26:34.664571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:34.664540 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 01:26:35.667328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:35.667298 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:26:35.667716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:35.667580 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:26:35.668221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:35.668205 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:26:44.219140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:44.219110 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h"] Oct 31 01:26:44.268130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:44.268100 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-r468h"] Oct 31 01:26:44.312099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:44.312061 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9546bc9c-e504-4985-ad38-f4a327ceb9eb" path="/var/lib/kubelet/pods/9546bc9c-e504-4985-ad38-f4a327ceb9eb/volumes" Oct 31 01:26:46.110264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:46.110228 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc"] Oct 31 01:26:46.110700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:46.110446 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" containerID="cri-o://c7429407961e146713ca2abfacc09284aff9920c72899abf5889de776b440b1f" gracePeriod=30 Oct 31 01:26:46.159317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:46.159283 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn"] Oct 31 01:26:46.159554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:46.159509 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" containerID="cri-o://fc088831af14fcad6042d69255b999fe9a145ee0c554bbc71021af22e901d10e" gracePeriod=30 Oct 31 01:26:51.108630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.108578 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:26:51.109047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.108639 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:26:51.354662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.354410 5756 generic.go:356] "Generic (PLEG): container finished" podID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerID="fc088831af14fcad6042d69255b999fe9a145ee0c554bbc71021af22e901d10e" exitCode=0 Oct 31 01:26:51.354662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.354586 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" event={"ID":"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d","Type":"ContainerDied","Data":"fc088831af14fcad6042d69255b999fe9a145ee0c554bbc71021af22e901d10e"} Oct 31 01:26:51.357035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.357008 5756 generic.go:356] "Generic (PLEG): container finished" podID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerID="c7429407961e146713ca2abfacc09284aff9920c72899abf5889de776b440b1f" exitCode=0 Oct 31 01:26:51.357395 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.357375 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" event={"ID":"d3e0aac8-3fbf-4463-b067-22a0c2943388","Type":"ContainerDied","Data":"c7429407961e146713ca2abfacc09284aff9920c72899abf5889de776b440b1f"} Oct 31 01:26:51.460685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.460659 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:26:51.497707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.497678 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:26:51.567449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567417 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.567626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567492 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.567626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567518 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.567626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567547 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.567626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567587 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.567626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567609 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.567934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567637 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.567934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567680 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.567934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567828 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.567934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567848 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.567934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567907 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567942 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.567979 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568032 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568059 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568088 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568113 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568119 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568138 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568157 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.568209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568203 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.568910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568228 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token\") pod \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\" (UID: \"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d\") " Oct 31 01:26:51.568910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568328 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb\") pod \"d3e0aac8-3fbf-4463-b067-22a0c2943388\" (UID: \"d3e0aac8-3fbf-4463-b067-22a0c2943388\") " Oct 31 01:26:51.568910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568338 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.568910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568141 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.568910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568592 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:26:51.568910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.568818 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.569415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569392 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569410 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569422 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569439 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569455 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569469 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569485 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3e0aac8-3fbf-4463-b067-22a0c2943388-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569499 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569513 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.569962 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.569527 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.570396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.570345 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.571093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.571050 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:26:51.571194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.571119 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:26:51.571194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.571148 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "kube-api-access-nlfpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:51.571194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.571183 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "kube-api-access-bps8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:51.571688 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.571651 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:26:51.571843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.571828 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token" pod "d3e0aac8-3fbf-4463-b067-22a0c2943388" (UID: "d3e0aac8-3fbf-4463-b067-22a0c2943388"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:51.572386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.572368 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token" pod "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" (UID: "a6e3ab4a-624d-4f34-a4ff-08b42e3e550d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:26:51.671685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671601 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfpb\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-kube-api-access-nlfpb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671641 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671659 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671673 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671686 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671699 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3e0aac8-3fbf-4463-b067-22a0c2943388-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671717 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bps8c\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-kube-api-access-bps8c\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671732 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:51.671999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:51.671745 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:26:52.366352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.366284 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" Oct 31 01:26:52.366352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.366314 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc" event={"ID":"d3e0aac8-3fbf-4463-b067-22a0c2943388","Type":"ContainerDied","Data":"a0c4aa21f2d923789b029fdaa3d7b30a06112d88201649fd116a80e55e23e759"} Oct 31 01:26:52.366897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.366388 5756 scope.go:122] "RemoveContainer" containerID="c7429407961e146713ca2abfacc09284aff9920c72899abf5889de776b440b1f" Oct 31 01:26:52.368447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.368411 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" Oct 31 01:26:52.368595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.368313 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn" event={"ID":"a6e3ab4a-624d-4f34-a4ff-08b42e3e550d","Type":"ContainerDied","Data":"db66c4abdd2bf00c914250212f868fcf1fee10f3a55a3d9a343e64a8d1adf08a"} Oct 31 01:26:52.378067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.378050 5756 scope.go:122] "RemoveContainer" containerID="fc088831af14fcad6042d69255b999fe9a145ee0c554bbc71021af22e901d10e" Oct 31 01:26:52.534289 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.534253 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc"] Oct 31 01:26:52.583299 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.583266 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g7whc"] Oct 31 01:26:52.677752 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.676950 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn"] Oct 31 01:26:52.696104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.696070 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-7hkrn"] Oct 31 01:26:52.938985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:52.938902 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:26:54.311817 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:54.311787 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" path="/var/lib/kubelet/pods/a6e3ab4a-624d-4f34-a4ff-08b42e3e550d/volumes" Oct 31 01:26:54.312406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:26:54.312389 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" path="/var/lib/kubelet/pods/d3e0aac8-3fbf-4463-b067-22a0c2943388/volumes" Oct 31 01:27:00.045818 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:00.045778 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn"] Oct 31 01:27:00.046307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:00.046052 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" containerID="cri-o://a18c48dc251f214495636c773539f193aedaf3b01c183d8701111cffa90ef374" gracePeriod=30 Oct 31 01:27:02.067328 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:02.067280 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:27:02.499708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:02.499629 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm"] Oct 31 01:27:02.499918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:02.499895 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" containerID="cri-o://383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539" gracePeriod=30 Oct 31 01:27:03.450627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:03.450548 5756 generic.go:356] "Generic (PLEG): container finished" podID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerID="f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86" exitCode=0 Oct 31 01:27:03.451033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:03.450618 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" event={"ID":"c1200043-f2f0-45c1-8620-ac65307b0bdc","Type":"ContainerDied","Data":"f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86"} Oct 31 01:27:03.451173 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:03.451146 5756 scope.go:122] "RemoveContainer" containerID="f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86" Oct 31 01:27:03.452733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:03.452701 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:27:03.452850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:03.452771 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:27:03.452850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:03.452802 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:27:04.462051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:04.462019 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" event={"ID":"c1200043-f2f0-45c1-8620-ac65307b0bdc","Type":"ContainerStarted","Data":"f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec"} Oct 31 01:27:05.017375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.017343 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:27:05.017719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.017606 5756 patch_prober.go:29] interesting pod/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:27:05.017911 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.017857 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 01:27:05.092754 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.092693 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq"] Oct 31 01:27:05.097881 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.097807 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4bcda6-tjtkq"] Oct 31 01:27:05.470763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.470731 5756 generic.go:356] "Generic (PLEG): container finished" podID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerID="a18c48dc251f214495636c773539f193aedaf3b01c183d8701111cffa90ef374" exitCode=0 Oct 31 01:27:05.471276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.470802 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" event={"ID":"42112337-beed-46a8-b4d6-3f69f46c68f1","Type":"ContainerDied","Data":"a18c48dc251f214495636c773539f193aedaf3b01c183d8701111cffa90ef374"} Oct 31 01:27:05.800327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.800306 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:27:05.899983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.899947 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900093 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900142 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900219 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900316 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900354 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900370 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:05.900465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900436 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900469 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900572 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo\") pod \"42112337-beed-46a8-b4d6-3f69f46c68f1\" (UID: \"42112337-beed-46a8-b4d6-3f69f46c68f1\") " Oct 31 01:27:05.900668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900609 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:05.900668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900618 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:05.900764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900686 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:05.900764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.900751 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:05.901269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.901243 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:05.901408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.901274 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42112337-beed-46a8-b4d6-3f69f46c68f1-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:05.901408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.901289 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:05.901408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.901304 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:05.901408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.901318 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:05.902409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.902388 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:05.902692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.902678 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "kube-api-access-fx76t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:05.902779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.902764 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:05.902985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:05.902964 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo" pod "42112337-beed-46a8-b4d6-3f69f46c68f1" (UID: "42112337-beed-46a8-b4d6-3f69f46c68f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:27:06.002335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.002303 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:06.002335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.002335 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:06.002543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.002348 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx76t\" (UniqueName: \"kubernetes.io/projected/42112337-beed-46a8-b4d6-3f69f46c68f1-kube-api-access-fx76t\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:06.002543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.002360 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42112337-beed-46a8-b4d6-3f69f46c68f1-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:06.017566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.017539 5756 patch_prober.go:29] interesting pod/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:27:06.017720 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.017586 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 01:27:06.312052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.312011 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9697036c-a4d8-425f-ac2d-837e3cd29400" path="/var/lib/kubelet/pods/9697036c-a4d8-425f-ac2d-837e3cd29400/volumes" Oct 31 01:27:06.478932 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.478902 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" Oct 31 01:27:06.479366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.478907 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn" event={"ID":"42112337-beed-46a8-b4d6-3f69f46c68f1","Type":"ContainerDied","Data":"fc3ce3974292e2b8a71b4faeb90568cb65a4ae5cbe3bd9cb9a2e4727e2e15b6f"} Oct 31 01:27:06.479366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.479036 5756 scope.go:122] "RemoveContainer" containerID="a18c48dc251f214495636c773539f193aedaf3b01c183d8701111cffa90ef374" Oct 31 01:27:06.665120 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.665037 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn"] Oct 31 01:27:06.667736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.667716 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-br6xn"] Oct 31 01:27:06.981505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.981397 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:27:06.981505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:06.981475 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:27:07.018274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.018232 5756 patch_prober.go:29] interesting pod/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:27:07.018464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.018298 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 01:27:07.811274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.811250 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:27:07.953944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.953513 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.953944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.953577 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.953944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.953648 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.953944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.953680 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.953944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.953842 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:07.954462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.953998 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.954462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954033 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.954462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954071 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.954462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954139 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.954462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954220 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo\") pod \"2087da4a-c455-430b-81cc-7c291549f3ab\" (UID: \"2087da4a-c455-430b-81cc-7c291549f3ab\") " Oct 31 01:27:07.954714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954562 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:07.955101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954843 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:07.955101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.954891 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2087da4a-c455-430b-81cc-7c291549f3ab-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:07.955101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.955069 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:07.955647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.955206 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:07.955975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.955954 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:07.956452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.956427 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "kube-api-access-ffwtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:07.956991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.956972 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:27:07.957900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.957850 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:07.958314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:07.958286 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy" pod "2087da4a-c455-430b-81cc-7c291549f3ab" (UID: "2087da4a-c455-430b-81cc-7c291549f3ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:27:08.018415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.018372 5756 patch_prober.go:29] interesting pod/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:27:08.018603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.018430 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058277 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058314 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058329 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058344 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2087da4a-c455-430b-81cc-7c291549f3ab-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058358 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2087da4a-c455-430b-81cc-7c291549f3ab-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058373 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.058466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.058387 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffwtn\" (UniqueName: \"kubernetes.io/projected/2087da4a-c455-430b-81cc-7c291549f3ab-kube-api-access-ffwtn\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:08.062992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.062933 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs"] Oct 31 01:27:08.067719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.067692 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6450408c-mpzzs"] Oct 31 01:27:08.605180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.605147 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" path="/var/lib/kubelet/pods/42112337-beed-46a8-b4d6-3f69f46c68f1/volumes" Oct 31 01:27:08.606198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.606158 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4434f2-0759-4f3d-adaf-20a0ed8d37c7" path="/var/lib/kubelet/pods/8d4434f2-0759-4f3d-adaf-20a0ed8d37c7/volumes" Oct 31 01:27:08.616281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.616243 5756 generic.go:356] "Generic (PLEG): container finished" podID="2087da4a-c455-430b-81cc-7c291549f3ab" containerID="383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539" exitCode=0 Oct 31 01:27:08.616819 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.616801 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" Oct 31 01:27:08.618957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.618928 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" event={"ID":"2087da4a-c455-430b-81cc-7c291549f3ab","Type":"ContainerDied","Data":"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539"} Oct 31 01:27:08.619200 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.619178 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm" event={"ID":"2087da4a-c455-430b-81cc-7c291549f3ab","Type":"ContainerDied","Data":"b75d9f3b4b0ec83234b51547e74734046802f7e0b632a1e98ccf389a1f54b123"} Oct 31 01:27:08.619371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.619319 5756 scope.go:122] "RemoveContainer" containerID="383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539" Oct 31 01:27:08.662317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.662287 5756 scope.go:122] "RemoveContainer" containerID="383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539" Oct 31 01:27:08.671141 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:08.670700 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539\": container with ID starting with 383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539 not found: ID does not exist" containerID="383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539" Oct 31 01:27:08.671141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.670738 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539"} err="failed to get container status \"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539\": rpc error: code = NotFound desc = could not find container \"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539\": container with ID starting with 383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539 not found: ID does not exist" Oct 31 01:27:08.677277 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:08.677247 5756 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539\": container with ID starting with 383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539 not found: ID does not exist" Oct 31 01:27:08.678474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.678448 5756 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539\": container with ID starting with 383001e429dc0fe83ed5f0bae2a8325a71b0cc5c35e475896e6ed7987fa72539 not found: ID does not exist" Oct 31 01:27:08.790167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.790138 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm"] Oct 31 01:27:08.797442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.797405 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-srmqm"] Oct 31 01:27:08.997660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:08.997581 5756 scope.go:122] "RemoveContainer" containerID="0d8d8ad7cd408d72c39120d23bad8234b64a6040de63416b25724dc7a58a89f3" Oct 31 01:27:09.009797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.009776 5756 scope.go:122] "RemoveContainer" containerID="fb34516e7f163b3ff5e4e5fc64f2597bbe4a176f79189d5d56be7e457e1bc630" Oct 31 01:27:09.018245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.018216 5756 patch_prober.go:29] interesting pod/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 31 01:27:09.018372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.018282 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 31 01:27:09.023391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.023366 5756 scope.go:122] "RemoveContainer" containerID="60a3a8f0748c570664b9615411f386be346ec09ba911e4e92f338d87864ae5c8" Oct 31 01:27:09.060915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.059051 5756 scope.go:122] "RemoveContainer" containerID="acdbd4fc5e7023a9fd4c799c6fea2e8aa1d0a3c98e7f1aa484ab07b8066e26d0" Oct 31 01:27:09.072547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.072525 5756 scope.go:122] "RemoveContainer" containerID="b7839b54a5099cfe8e378b87dcb66d28d0870ccb251931c4627085bdfd0c5615" Oct 31 01:27:09.082327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:09.082304 5756 scope.go:122] "RemoveContainer" containerID="08f332103549786e711287db222ccac79a5ea12b304af3ef1ac7850e53918d5e" Oct 31 01:27:10.031789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:10.031655 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:27:10.033023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:10.032952 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:27:10.034895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:10.034836 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:27:10.314030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:10.313951 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" path="/var/lib/kubelet/pods/2087da4a-c455-430b-81cc-7c291549f3ab/volumes" Oct 31 01:27:16.293164 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:16.292962 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:22.217591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:22.217560 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:27:23.611267 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:23.611234 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:26.307974 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:26.307940 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:32.066733 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:32.066696 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:27:32.807516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.807476 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr"] Oct 31 01:27:32.808049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808031 5756 cpu_manager.go:414] "containerMap: removing container" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" Oct 31 01:27:32.808049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808046 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808066 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9697036c-a4d8-425f-ac2d-837e3cd29400" containerName="revoke-keys" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808072 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9697036c-a4d8-425f-ac2d-837e3cd29400" containerName="revoke-keys" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808082 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808088 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808096 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808100 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808110 5756 cpu_manager.go:414] "containerMap: removing container" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808114 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808136 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6f242e15-32d2-4f0e-b93f-f77993db4169" containerName="revoke-keys" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808141 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f242e15-32d2-4f0e-b93f-f77993db4169" containerName="revoke-keys" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808155 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808159 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" Oct 31 01:27:32.808234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808242 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9697036c-a4d8-425f-ac2d-837e3cd29400" containerName="revoke-keys" Oct 31 01:27:32.808904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808252 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3955cd72-831e-4360-a2b7-bef5dd3f36ea" containerName="istio-proxy" Oct 31 01:27:32.808904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808258 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2087da4a-c455-430b-81cc-7c291549f3ab" containerName="istio-proxy" Oct 31 01:27:32.808904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808265 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e3ab4a-624d-4f34-a4ff-08b42e3e550d" containerName="istio-proxy" Oct 31 01:27:32.808904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808275 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f242e15-32d2-4f0e-b93f-f77993db4169" containerName="revoke-keys" Oct 31 01:27:32.808904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808283 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e0aac8-3fbf-4463-b067-22a0c2943388" containerName="istio-proxy" Oct 31 01:27:32.808904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.808290 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="42112337-beed-46a8-b4d6-3f69f46c68f1" containerName="istio-proxy" Oct 31 01:27:32.811163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.811141 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:32.872955 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.872923 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr"] Oct 31 01:27:32.899600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.899560 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:32.899809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.899749 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2s6\" (UniqueName: \"kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:32.899912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.899899 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:32.917826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.917793 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:27:32.917999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:32.917857 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:27:33.001609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.001570 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.001777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.001682 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.001819 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.001776 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2s6\" (UniqueName: \"kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.002337 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.002311 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.004087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.004051 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.060924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.060842 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2s6\" (UniqueName: \"kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6\") pod \"maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.124464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.124434 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:33.501966 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.501940 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr"] Oct 31 01:27:33.503538 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:27:33.503505 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6d1875_fc7b_4f74_a33c_c62c9097d6c7.slice/crio-f6c9274aef07c8f23c83c73ba0b9a3ee48389cc62ead5429ba2d5755f6ff4906 WatchSource:0}: Error finding container f6c9274aef07c8f23c83c73ba0b9a3ee48389cc62ead5429ba2d5755f6ff4906: Status 404 returned error can't find the container with id f6c9274aef07c8f23c83c73ba0b9a3ee48389cc62ead5429ba2d5755f6ff4906 Oct 31 01:27:33.775390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:33.775359 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" event={"ID":"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7","Type":"ContainerStarted","Data":"f6c9274aef07c8f23c83c73ba0b9a3ee48389cc62ead5429ba2d5755f6ff4906"} Oct 31 01:27:34.795269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:34.795235 5756 generic.go:356] "Generic (PLEG): container finished" podID="5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" containerID="8f81f27a725a3a9e529fb4a1976fe4cbe45479dbfb9a7ae0dc2b64275169c4ce" exitCode=0 Oct 31 01:27:34.795722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:34.795283 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" event={"ID":"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7","Type":"ContainerDied","Data":"8f81f27a725a3a9e529fb4a1976fe4cbe45479dbfb9a7ae0dc2b64275169c4ce"} Oct 31 01:27:36.004900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.004875 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:36.043122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.043096 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6\" (UniqueName: \"kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6\") pod \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " Oct 31 01:27:36.043279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.043165 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca\") pod \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " Oct 31 01:27:36.043279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.043212 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca\") pod \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\" (UID: \"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7\") " Oct 31 01:27:36.043602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.043577 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca" pod "5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" (UID: "5c6d1875-fc7b-4f74-a33c-c62c9097d6c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:27:36.043794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.043740 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:36.045382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.045356 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca" pod "5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" (UID: "5c6d1875-fc7b-4f74-a33c-c62c9097d6c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:27:36.045382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.045364 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6" pod "5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" (UID: "5c6d1875-fc7b-4f74-a33c-c62c9097d6c7"). InnerVolumeSpecName "kube-api-access-tn2s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:27:36.145350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.145312 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:36.145350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.145351 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn2s6\" (UniqueName: \"kubernetes.io/projected/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7-kube-api-access-tn2s6\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:27:36.382455 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:36.382414 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:36.808920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.808883 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" event={"ID":"5c6d1875-fc7b-4f74-a33c-c62c9097d6c7","Type":"ContainerDied","Data":"f6c9274aef07c8f23c83c73ba0b9a3ee48389cc62ead5429ba2d5755f6ff4906"} Oct 31 01:27:36.808920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.808911 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr" Oct 31 01:27:36.809118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:27:36.808932 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6c9274aef07c8f23c83c73ba0b9a3ee48389cc62ead5429ba2d5755f6ff4906" Oct 31 01:27:38.540713 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:38.540673 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:46.471770 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:46.471728 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:53.609532 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:53.609492 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:56.489483 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:56.489448 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:58.261341 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:58.261303 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:27:58.261835 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:27:58.261774 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:28:01.050982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:01.050946 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r"] Oct 31 01:28:01.098544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:01.098501 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33e221b4-tq54r"] Oct 31 01:28:02.066300 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:28:02.066260 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:28:02.312029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:02.311995 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06dd1f99-c151-4b46-9853-eab6054ef5da" path="/var/lib/kubelet/pods/06dd1f99-c151-4b46-9853-eab6054ef5da/volumes" Oct 31 01:28:06.560005 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:28:06.559964 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087da4a_c455_430b_81cc_7c291549f3ab.slice\": RecentStats: unable to find data in memory cache]" Oct 31 01:28:07.060832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.060800 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np"] Oct 31 01:28:07.061425 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.061409 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" containerName="revoke-keys" Oct 31 01:28:07.061473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.061428 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" containerName="revoke-keys" Oct 31 01:28:07.061561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.061552 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" containerName="revoke-keys" Oct 31 01:28:07.065110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.065091 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.122015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.121983 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:07.122356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.122336 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:28:07.123704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.123678 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np"] Oct 31 01:28:07.172105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.172075 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:28:07.176136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.176109 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrfw\" (UniqueName: \"kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.176235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.176187 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.176274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.176243 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.277377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.277339 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrfw\" (UniqueName: \"kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.277559 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.277430 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.277559 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.277451 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.278156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.278133 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.280165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.280143 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.339624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.339568 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrfw\" (UniqueName: \"kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw\") pod \"maas-api-revoke-keys-e2e-api-a-f57e85-rr8np\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.378641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.378616 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:07.782925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:07.782895 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np"] Oct 31 01:28:07.784111 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:28:07.784081 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af2ac60_f40f_46d1_ad21_b98a0b080979.slice/crio-1241246bdfd000362443cd4e142d95627068bf4d8274a3f379be4c05d5e3ea74 WatchSource:0}: Error finding container 1241246bdfd000362443cd4e142d95627068bf4d8274a3f379be4c05d5e3ea74: Status 404 returned error can't find the container with id 1241246bdfd000362443cd4e142d95627068bf4d8274a3f379be4c05d5e3ea74 Oct 31 01:28:08.018434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:08.018401 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" event={"ID":"0af2ac60-f40f-46d1-ad21-b98a0b080979","Type":"ContainerStarted","Data":"1241246bdfd000362443cd4e142d95627068bf4d8274a3f379be4c05d5e3ea74"} Oct 31 01:28:09.026639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:09.026538 5756 generic.go:356] "Generic (PLEG): container finished" podID="0af2ac60-f40f-46d1-ad21-b98a0b080979" containerID="974aae633f78ab7508e30b80f8dbe915c17a12ed2b935bf1a57bc6aa02a570cb" exitCode=0 Oct 31 01:28:09.027034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:09.026626 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" event={"ID":"0af2ac60-f40f-46d1-ad21-b98a0b080979","Type":"ContainerDied","Data":"974aae633f78ab7508e30b80f8dbe915c17a12ed2b935bf1a57bc6aa02a570cb"} Oct 31 01:28:09.460800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:09.460767 5756 scope.go:122] "RemoveContainer" containerID="1efd019ec67e92a8b7f4c9feb8f0450e5ce66b61b1f40414cfeca731fa612b66" Oct 31 01:28:09.470540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:09.470520 5756 scope.go:122] "RemoveContainer" containerID="11cc66591ca179bbde99634f879c33a2046b511741b6ab4dc29422b5899425c5" Oct 31 01:28:10.227322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.227295 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:10.329364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.329286 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca\") pod \"0af2ac60-f40f-46d1-ad21-b98a0b080979\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " Oct 31 01:28:10.329364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.329324 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca\") pod \"0af2ac60-f40f-46d1-ad21-b98a0b080979\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " Oct 31 01:28:10.329586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.329532 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw\" (UniqueName: \"kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw\") pod \"0af2ac60-f40f-46d1-ad21-b98a0b080979\" (UID: \"0af2ac60-f40f-46d1-ad21-b98a0b080979\") " Oct 31 01:28:10.329739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.329711 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca" pod "0af2ac60-f40f-46d1-ad21-b98a0b080979" (UID: "0af2ac60-f40f-46d1-ad21-b98a0b080979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:10.330061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.330041 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:10.331589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.331566 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw" pod "0af2ac60-f40f-46d1-ad21-b98a0b080979" (UID: "0af2ac60-f40f-46d1-ad21-b98a0b080979"). InnerVolumeSpecName "kube-api-access-6xrfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:10.331686 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.331605 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca" pod "0af2ac60-f40f-46d1-ad21-b98a0b080979" (UID: "0af2ac60-f40f-46d1-ad21-b98a0b080979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:10.431315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.431288 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xrfw\" (UniqueName: \"kubernetes.io/projected/0af2ac60-f40f-46d1-ad21-b98a0b080979-kube-api-access-6xrfw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:10.431315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:10.431314 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af2ac60-f40f-46d1-ad21-b98a0b080979-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:11.054047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:11.054018 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" Oct 31 01:28:11.054214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:11.054072 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np" event={"ID":"0af2ac60-f40f-46d1-ad21-b98a0b080979","Type":"ContainerDied","Data":"1241246bdfd000362443cd4e142d95627068bf4d8274a3f379be4c05d5e3ea74"} Oct 31 01:28:11.054214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:11.054163 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1241246bdfd000362443cd4e142d95627068bf4d8274a3f379be4c05d5e3ea74" Oct 31 01:28:12.045617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:12.045587 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf"] Oct 31 01:28:12.094724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:12.094700 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1383412-njklf"] Oct 31 01:28:12.311619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:12.311541 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aac2a03-80c5-4f70-b09b-c72312711a86" path="/var/lib/kubelet/pods/9aac2a03-80c5-4f70-b09b-c72312711a86/volumes" Oct 31 01:28:32.066116 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:28:32.066073 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:28:41.595286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:41.595255 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:28:48.015815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:48.015736 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f"] Oct 31 01:28:48.016288 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:48.015946 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" containerID="cri-o://18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15" gracePeriod=30 Oct 31 01:28:48.065297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:48.065267 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9"] Oct 31 01:28:48.065495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:48.065478 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" containerID="cri-o://3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145" gracePeriod=30 Oct 31 01:28:49.721321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.721288 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf"] Oct 31 01:28:49.721856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.721841 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0af2ac60-f40f-46d1-ad21-b98a0b080979" containerName="revoke-keys" Oct 31 01:28:49.721856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.721857 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af2ac60-f40f-46d1-ad21-b98a0b080979" containerName="revoke-keys" Oct 31 01:28:49.721989 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.721980 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af2ac60-f40f-46d1-ad21-b98a0b080979" containerName="revoke-keys" Oct 31 01:28:49.725721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.725697 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.782898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.782855 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf"] Oct 31 01:28:49.793477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.793444 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.793627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.793523 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqqx2\" (UniqueName: \"kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.793739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.793723 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.828341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.828318 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:28:49.828484 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.828433 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:28:49.895658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.895629 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.895795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.895731 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.895837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.895796 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqqx2\" (UniqueName: \"kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.896441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.896397 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.898219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.898197 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:49.952320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:49.952290 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqqx2\" (UniqueName: \"kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2\") pod \"maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:50.039514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:50.039488 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:50.415394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:50.415364 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf"] Oct 31 01:28:50.416406 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:28:50.416379 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6cd153c_292b_4404_b7fe_38b44a59648b.slice/crio-4d73cd10462f1ef402f04fe4c0ec542162e96347689ca3709a311efd781e2ac2 WatchSource:0}: Error finding container 4d73cd10462f1ef402f04fe4c0ec542162e96347689ca3709a311efd781e2ac2: Status 404 returned error can't find the container with id 4d73cd10462f1ef402f04fe4c0ec542162e96347689ca3709a311efd781e2ac2 Oct 31 01:28:51.371253 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:51.371210 5756 generic.go:356] "Generic (PLEG): container finished" podID="c6cd153c-292b-4404-b7fe-38b44a59648b" containerID="7e1ca0d48e3c865cee78f6ccb30aede49dcdad2856a640e30832f2751affeaa5" exitCode=0 Oct 31 01:28:51.371671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:51.371290 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" event={"ID":"c6cd153c-292b-4404-b7fe-38b44a59648b","Type":"ContainerDied","Data":"7e1ca0d48e3c865cee78f6ccb30aede49dcdad2856a640e30832f2751affeaa5"} Oct 31 01:28:51.371671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:51.371336 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" event={"ID":"c6cd153c-292b-4404-b7fe-38b44a59648b","Type":"ContainerStarted","Data":"4d73cd10462f1ef402f04fe4c0ec542162e96347689ca3709a311efd781e2ac2"} Oct 31 01:28:52.553614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.553588 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:52.637359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.637279 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca\") pod \"c6cd153c-292b-4404-b7fe-38b44a59648b\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " Oct 31 01:28:52.637507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.637479 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2\" (UniqueName: \"kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2\") pod \"c6cd153c-292b-4404-b7fe-38b44a59648b\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " Oct 31 01:28:52.637562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.637526 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca\") pod \"c6cd153c-292b-4404-b7fe-38b44a59648b\" (UID: \"c6cd153c-292b-4404-b7fe-38b44a59648b\") " Oct 31 01:28:52.637958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.637931 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca" pod "c6cd153c-292b-4404-b7fe-38b44a59648b" (UID: "c6cd153c-292b-4404-b7fe-38b44a59648b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:52.639524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.639506 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca" pod "c6cd153c-292b-4404-b7fe-38b44a59648b" (UID: "c6cd153c-292b-4404-b7fe-38b44a59648b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:28:52.639644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.639622 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2" pod "c6cd153c-292b-4404-b7fe-38b44a59648b" (UID: "c6cd153c-292b-4404-b7fe-38b44a59648b"). InnerVolumeSpecName "kube-api-access-nqqx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:52.739699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.739666 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqqx2\" (UniqueName: \"kubernetes.io/projected/c6cd153c-292b-4404-b7fe-38b44a59648b-kube-api-access-nqqx2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:52.739699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.739695 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:52.739699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:52.739706 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6cd153c-292b-4404-b7fe-38b44a59648b-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.338907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.338880 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:28:53.370233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.370208 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:28:53.387564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.387536 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" Oct 31 01:28:53.387564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.387556 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf" event={"ID":"c6cd153c-292b-4404-b7fe-38b44a59648b","Type":"ContainerDied","Data":"4d73cd10462f1ef402f04fe4c0ec542162e96347689ca3709a311efd781e2ac2"} Oct 31 01:28:53.387738 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.387601 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d73cd10462f1ef402f04fe4c0ec542162e96347689ca3709a311efd781e2ac2" Oct 31 01:28:53.393421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.393395 5756 generic.go:356] "Generic (PLEG): container finished" podID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerID="18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15" exitCode=0 Oct 31 01:28:53.393538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.393486 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" event={"ID":"0429a525-32e3-45a1-aecd-829241e2ebd8","Type":"ContainerDied","Data":"18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15"} Oct 31 01:28:53.393538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.393518 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" event={"ID":"0429a525-32e3-45a1-aecd-829241e2ebd8","Type":"ContainerDied","Data":"0a134f9054ea4de5e9eaed56d161be5b587c543292f3b1046980932552c5aef0"} Oct 31 01:28:53.393617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.393540 5756 scope.go:122] "RemoveContainer" containerID="18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15" Oct 31 01:28:53.393650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.393631 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f" Oct 31 01:28:53.395444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.395419 5756 generic.go:356] "Generic (PLEG): container finished" podID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerID="3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145" exitCode=0 Oct 31 01:28:53.395556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.395456 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" event={"ID":"85239cc5-b9d9-4237-b649-d27d998c9beb","Type":"ContainerDied","Data":"3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145"} Oct 31 01:28:53.395556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.395466 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" Oct 31 01:28:53.395556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.395493 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9" event={"ID":"85239cc5-b9d9-4237-b649-d27d998c9beb","Type":"ContainerDied","Data":"802956e99c7ab5f2e0d6b64b424cd9adf09e3f2fdb1b482a84508345f0368be0"} Oct 31 01:28:53.403643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.403628 5756 scope.go:122] "RemoveContainer" containerID="18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15" Oct 31 01:28:53.403919 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:28:53.403896 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15\": container with ID starting with 18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15 not found: ID does not exist" containerID="18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15" Oct 31 01:28:53.403973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.403932 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15"} err="failed to get container status \"18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15\": rpc error: code = NotFound desc = could not find container \"18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15\": container with ID starting with 18ad6d079a3f9d64c9aeaa02b9f4c016668537527bbd018e22ea8b4572701a15 not found: ID does not exist" Oct 31 01:28:53.403973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.403953 5756 scope.go:122] "RemoveContainer" containerID="3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145" Oct 31 01:28:53.412709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.412682 5756 scope.go:122] "RemoveContainer" containerID="3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145" Oct 31 01:28:53.412999 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:28:53.412980 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145\": container with ID starting with 3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145 not found: ID does not exist" containerID="3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145" Oct 31 01:28:53.413055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.413010 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145"} err="failed to get container status \"3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145\": rpc error: code = NotFound desc = could not find container \"3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145\": container with ID starting with 3d304edcad2d5e6efb38191ef19239ac9e150f8a7fd7b7126ccadf3701a7f145 not found: ID does not exist" Oct 31 01:28:53.449339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449317 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.449476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449356 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449400 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449449 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.449639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449498 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.449639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449515 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.449639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449589 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449653 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.449744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449704 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449735 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449759 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449789 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449823 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.449916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449834 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.449916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449906 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:53.450183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449978 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket\") pod \"0429a525-32e3-45a1-aecd-829241e2ebd8\" (UID: \"0429a525-32e3-45a1-aecd-829241e2ebd8\") " Oct 31 01:28:53.450183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450014 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.450183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450035 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.450183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450066 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.450183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450163 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.450183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.449771 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450192 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket\") pod \"85239cc5-b9d9-4237-b649-d27d998c9beb\" (UID: \"85239cc5-b9d9-4237-b649-d27d998c9beb\") " Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450359 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450601 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450618 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450815 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.450875 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451133 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451154 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451176 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451191 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451205 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85239cc5-b9d9-4237-b649-d27d998c9beb-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451218 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0429a525-32e3-45a1-aecd-829241e2ebd8-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451232 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451246 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451259 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.451644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.451272 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.452648 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.452619 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "kube-api-access-5fzjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:53.452768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.452744 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:28:53.452925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.452903 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "kube-api-access-m247z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:53.453213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.453185 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.453213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.453191 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:53.453365 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.453270 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy" pod "0429a525-32e3-45a1-aecd-829241e2ebd8" (UID: "0429a525-32e3-45a1-aecd-829241e2ebd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:28:53.453426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.453410 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:28:53.454066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.454043 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo" pod "85239cc5-b9d9-4237-b649-d27d998c9beb" (UID: "85239cc5-b9d9-4237-b649-d27d998c9beb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:28:53.553082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553017 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fzjm\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-kube-api-access-5fzjm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553044 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553060 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553068 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553077 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m247z\" (UniqueName: \"kubernetes.io/projected/0429a525-32e3-45a1-aecd-829241e2ebd8-kube-api-access-m247z\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553085 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0429a525-32e3-45a1-aecd-829241e2ebd8-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553094 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553102 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85239cc5-b9d9-4237-b649-d27d998c9beb-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.553355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.553110 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0429a525-32e3-45a1-aecd-829241e2ebd8-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:28:53.819094 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.819015 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9"] Oct 31 01:28:53.826181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.826150 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-xbmg9"] Oct 31 01:28:53.934428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.934398 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f"] Oct 31 01:28:53.938196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:53.938169 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-8ml8f"] Oct 31 01:28:54.311203 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:54.311166 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" path="/var/lib/kubelet/pods/0429a525-32e3-45a1-aecd-829241e2ebd8/volumes" Oct 31 01:28:54.311679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:28:54.311665 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" path="/var/lib/kubelet/pods/85239cc5-b9d9-4237-b649-d27d998c9beb/volumes" Oct 31 01:29:02.076985 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:29:02.076941 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:29:05.879617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.879585 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7918-f3eb51"] Oct 31 01:29:05.880168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880153 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c6cd153c-292b-4404-b7fe-38b44a59648b" containerName="revoke-keys" Oct 31 01:29:05.880215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880169 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6cd153c-292b-4404-b7fe-38b44a59648b" containerName="revoke-keys" Oct 31 01:29:05.880215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880200 5756 cpu_manager.go:414] "containerMap: removing container" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" Oct 31 01:29:05.880215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880205 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" Oct 31 01:29:05.880304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880224 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" Oct 31 01:29:05.880304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880229 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" Oct 31 01:29:05.880370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880317 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0429a525-32e3-45a1-aecd-829241e2ebd8" containerName="istio-proxy" Oct 31 01:29:05.880370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880331 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6cd153c-292b-4404-b7fe-38b44a59648b" containerName="revoke-keys" Oct 31 01:29:05.880370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.880342 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="85239cc5-b9d9-4237-b649-d27d998c9beb" containerName="istio-proxy" Oct 31 01:29:05.883608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.883577 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:05.922492 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.922465 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vsx8\" (UniqueName: \"kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8\") pod \"test-curl-7918-f3eb51\" (UID: \"a15092b5-4b54-4cf2-bd44-407b72bc4a16\") " pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:05.937168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.937142 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-f3eb51"] Oct 31 01:29:05.982938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:05.982910 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vxhdm\"" Oct 31 01:29:06.024187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:06.024154 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vsx8\" (UniqueName: \"kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8\") pod \"test-curl-7918-f3eb51\" (UID: \"a15092b5-4b54-4cf2-bd44-407b72bc4a16\") " pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:06.080805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:06.080775 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vsx8\" (UniqueName: \"kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8\") pod \"test-curl-7918-f3eb51\" (UID: \"a15092b5-4b54-4cf2-bd44-407b72bc4a16\") " pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:06.195717 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:06.195638 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:06.569368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:06.569341 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-f3eb51"] Oct 31 01:29:06.571132 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:29:06.571102 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda15092b5_4b54_4cf2_bd44_407b72bc4a16.slice/crio-bdd4981efce0bdf190c7b73528beb942d02a43fd70abdd4350dd40aa8843959e WatchSource:0}: Error finding container bdd4981efce0bdf190c7b73528beb942d02a43fd70abdd4350dd40aa8843959e: Status 404 returned error can't find the container with id bdd4981efce0bdf190c7b73528beb942d02a43fd70abdd4350dd40aa8843959e Oct 31 01:29:07.486321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:07.486279 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-f3eb51" event={"ID":"a15092b5-4b54-4cf2-bd44-407b72bc4a16","Type":"ContainerStarted","Data":"f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649"} Oct 31 01:29:07.486703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:07.486331 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-f3eb51" event={"ID":"a15092b5-4b54-4cf2-bd44-407b72bc4a16","Type":"ContainerStarted","Data":"bdd4981efce0bdf190c7b73528beb942d02a43fd70abdd4350dd40aa8843959e"} Oct 31 01:29:07.598150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:07.598097 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7918-f3eb51" podStartSLOduration=2.59808377 podStartE2EDuration="2.59808377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:07.597717325 +0000 UTC m=+1919.404684868" watchObservedRunningTime="2026-10-31 01:29:07.59808377 +0000 UTC m=+1919.405051312" Oct 31 01:29:09.028012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.027969 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7918-f3eb51"] Oct 31 01:29:09.032382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.032353 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7918-f3eb51"] Oct 31 01:29:09.498594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.498556 5756 scope.go:122] "RemoveContainer" containerID="9d5ed4c7380f40e8384481cba8537098cb946cbd72b02e1333f6ed101d994f9f" Oct 31 01:29:09.499183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.499131 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7918-f3eb51" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" containerName="test-curl-7918-f3eb51" containerID="cri-o://f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649" gracePeriod=2 Oct 31 01:29:09.549711 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:29:09.549677 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-f3eb51\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:09.572108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.572074 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w"] Oct 31 01:29:09.572956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.572936 5756 cpu_manager.go:414] "containerMap: removing container" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" containerName="test-curl-7918-f3eb51" Oct 31 01:29:09.573002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.572961 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" containerName="test-curl-7918-f3eb51" Oct 31 01:29:09.573164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.573152 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" containerName="test-curl-7918-f3eb51" Oct 31 01:29:09.578591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.578575 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.630107 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:29:09.629700 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-f3eb51\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:09.633025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.632986 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w"] Oct 31 01:29:09.633176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.633085 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:09.633176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.633124 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:29:09.666684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.666650 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.666848 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.666804 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.666848 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.666825 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86b9n\" (UniqueName: \"kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.768186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.768108 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.768186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.768142 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86b9n\" (UniqueName: \"kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.768373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.768325 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.768940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.768917 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.770624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.770592 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.837639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.837600 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86b9n\" (UniqueName: \"kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n\") pod \"maas-api-revoke-keys-e2e-mt-f259852e-wcx7w\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:09.892300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:09.892264 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:10.266436 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:10.266396 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w"] Oct 31 01:29:10.267439 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:29:10.267408 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc39e123_28ad_4c99_9973_04ade69e2ae4.slice/crio-a8dd26098aa1d078a208314d242c738e514370847f6445cf8db7c0a8e837e78a WatchSource:0}: Error finding container a8dd26098aa1d078a208314d242c738e514370847f6445cf8db7c0a8e837e78a: Status 404 returned error can't find the container with id a8dd26098aa1d078a208314d242c738e514370847f6445cf8db7c0a8e837e78a Oct 31 01:29:10.507209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:10.507177 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" event={"ID":"cc39e123-28ad-4c99-9973-04ade69e2ae4","Type":"ContainerStarted","Data":"a8dd26098aa1d078a208314d242c738e514370847f6445cf8db7c0a8e837e78a"} Oct 31 01:29:11.514632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:11.514598 5756 generic.go:356] "Generic (PLEG): container finished" podID="cc39e123-28ad-4c99-9973-04ade69e2ae4" containerID="a8b91554d1dbda5ea93f0a7cc0f0b8dc863229727dd1c09c6e52c45d8f827c32" exitCode=0 Oct 31 01:29:11.515151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:11.514754 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" event={"ID":"cc39e123-28ad-4c99-9973-04ade69e2ae4","Type":"ContainerDied","Data":"a8b91554d1dbda5ea93f0a7cc0f0b8dc863229727dd1c09c6e52c45d8f827c32"} Oct 31 01:29:11.749929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:11.749900 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:11.795599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:11.795568 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8\" (UniqueName: \"kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8\") pod \"a15092b5-4b54-4cf2-bd44-407b72bc4a16\" (UID: \"a15092b5-4b54-4cf2-bd44-407b72bc4a16\") " Oct 31 01:29:11.797784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:11.797761 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8" pod "a15092b5-4b54-4cf2-bd44-407b72bc4a16" (UID: "a15092b5-4b54-4cf2-bd44-407b72bc4a16"). InnerVolumeSpecName "kube-api-access-8vsx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:11.897526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:11.897492 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vsx8\" (UniqueName: \"kubernetes.io/projected/a15092b5-4b54-4cf2-bd44-407b72bc4a16-kube-api-access-8vsx8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:12.311052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.311013 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" path="/var/lib/kubelet/pods/a15092b5-4b54-4cf2-bd44-407b72bc4a16/volumes" Oct 31 01:29:12.521596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.521555 5756 generic.go:356] "Generic (PLEG): container finished" podID="a15092b5-4b54-4cf2-bd44-407b72bc4a16" containerID="f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649" exitCode=137 Oct 31 01:29:12.522036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.521613 5756 scope.go:122] "RemoveContainer" containerID="f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649" Oct 31 01:29:12.522036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.521636 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-f3eb51" Oct 31 01:29:12.532878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.532843 5756 scope.go:122] "RemoveContainer" containerID="f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649" Oct 31 01:29:12.533162 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:29:12.533137 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649\": container with ID starting with f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649 not found: ID does not exist" containerID="f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649" Oct 31 01:29:12.533223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.533179 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649"} err="failed to get container status \"f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649\": rpc error: code = NotFound desc = could not find container \"f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649\": container with ID starting with f263196745180fb1af9314202c87194c72045acf491c7a8e0427cc18b8773649 not found: ID does not exist" Oct 31 01:29:12.686784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.686754 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:12.808483 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.807793 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca\") pod \"cc39e123-28ad-4c99-9973-04ade69e2ae4\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " Oct 31 01:29:12.808483 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.807953 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca\") pod \"cc39e123-28ad-4c99-9973-04ade69e2ae4\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " Oct 31 01:29:12.808483 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.808135 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n\" (UniqueName: \"kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n\") pod \"cc39e123-28ad-4c99-9973-04ade69e2ae4\" (UID: \"cc39e123-28ad-4c99-9973-04ade69e2ae4\") " Oct 31 01:29:12.809849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.809756 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca" pod "cc39e123-28ad-4c99-9973-04ade69e2ae4" (UID: "cc39e123-28ad-4c99-9973-04ade69e2ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:12.814025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.813992 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n" pod "cc39e123-28ad-4c99-9973-04ade69e2ae4" (UID: "cc39e123-28ad-4c99-9973-04ade69e2ae4"). InnerVolumeSpecName "kube-api-access-86b9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:12.820964 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.820935 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca" pod "cc39e123-28ad-4c99-9973-04ade69e2ae4" (UID: "cc39e123-28ad-4c99-9973-04ade69e2ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:12.910473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.910440 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:12.910473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.910472 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86b9n\" (UniqueName: \"kubernetes.io/projected/cc39e123-28ad-4c99-9973-04ade69e2ae4-kube-api-access-86b9n\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:12.910689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:12.910487 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc39e123-28ad-4c99-9973-04ade69e2ae4-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:13.529756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:13.529721 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" event={"ID":"cc39e123-28ad-4c99-9973-04ade69e2ae4","Type":"ContainerDied","Data":"a8dd26098aa1d078a208314d242c738e514370847f6445cf8db7c0a8e837e78a"} Oct 31 01:29:13.529756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:13.529741 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w" Oct 31 01:29:13.530179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:13.529769 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8dd26098aa1d078a208314d242c738e514370847f6445cf8db7c0a8e837e78a" Oct 31 01:29:18.456776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:18.456739 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6"] Oct 31 01:29:18.457294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:18.456994 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" containerID="cri-o://a0e04691ff32760ff39e36164d1cd729eb85e1d8a88074ccf412eb62cbb328e3" gracePeriod=30 Oct 31 01:29:20.946208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:20.946166 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:29:20.946615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:20.946233 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:29:23.599098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.599050 5756 generic.go:356] "Generic (PLEG): container finished" podID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerID="a0e04691ff32760ff39e36164d1cd729eb85e1d8a88074ccf412eb62cbb328e3" exitCode=0 Oct 31 01:29:23.599475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.599125 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" event={"ID":"21f2323d-4169-4e62-a62d-a8a3e2d1f595","Type":"ContainerDied","Data":"a0e04691ff32760ff39e36164d1cd729eb85e1d8a88074ccf412eb62cbb328e3"} Oct 31 01:29:23.727708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.727652 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:29:23.864501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864473 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864543 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864591 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864613 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864644 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864668 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864696 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864728 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:23.864954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864751 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.864954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864801 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket\") pod \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\" (UID: \"21f2323d-4169-4e62-a62d-a8a3e2d1f595\") " Oct 31 01:29:23.865167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.864990 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:23.865167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865002 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:23.865167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865040 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:23.865327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865308 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:23.865548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865525 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.865548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865549 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.865757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865560 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.865757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865569 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.865757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.865577 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.867171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.867137 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:29:23.867171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.867154 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "kube-api-access-qbq74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:23.867171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.867167 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:23.867802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.867784 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo" pod "21f2323d-4169-4e62-a62d-a8a3e2d1f595" (UID: "21f2323d-4169-4e62-a62d-a8a3e2d1f595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:29:23.966878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.966836 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.967033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.966898 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbq74\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-kube-api-access-qbq74\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.967033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.966910 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:23.967033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:23.966919 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f2323d-4169-4e62-a62d-a8a3e2d1f595-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:24.045986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.045957 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh"] Oct 31 01:29:24.048468 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.048446 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ce49-rn8fh"] Oct 31 01:29:24.313055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.312980 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29740221-25b6-4367-857f-67a9703c933e" path="/var/lib/kubelet/pods/29740221-25b6-4367-857f-67a9703c933e/volumes" Oct 31 01:29:24.606324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.606253 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" Oct 31 01:29:24.606723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.606256 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6" event={"ID":"21f2323d-4169-4e62-a62d-a8a3e2d1f595","Type":"ContainerDied","Data":"af354c40af44f6d2c821a018467f600a88afdbe6781387288b5d875f1f881b2d"} Oct 31 01:29:24.606723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.606384 5756 scope.go:122] "RemoveContainer" containerID="a0e04691ff32760ff39e36164d1cd729eb85e1d8a88074ccf412eb62cbb328e3" Oct 31 01:29:24.820375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.820344 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6"] Oct 31 01:29:24.869610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:24.869541 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-hvtw6"] Oct 31 01:29:26.311736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:26.311702 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" path="/var/lib/kubelet/pods/21f2323d-4169-4e62-a62d-a8a3e2d1f595/volumes" Oct 31 01:29:32.067544 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:29:32.067504 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:29:39.181054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:39.181020 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr"] Oct 31 01:29:39.230619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:39.230582 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3f72e4-8jvqr"] Oct 31 01:29:40.312279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:40.312244 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6d1875-fc7b-4f74-a33c-c62c9097d6c7" path="/var/lib/kubelet/pods/5c6d1875-fc7b-4f74-a33c-c62c9097d6c7/volumes" Oct 31 01:29:41.609376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.609337 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr"] Oct 31 01:29:41.610112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.610090 5756 cpu_manager.go:414] "containerMap: removing container" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" Oct 31 01:29:41.610226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.610113 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" Oct 31 01:29:41.610226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.610174 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cc39e123-28ad-4c99-9973-04ade69e2ae4" containerName="revoke-keys" Oct 31 01:29:41.610226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.610183 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc39e123-28ad-4c99-9973-04ade69e2ae4" containerName="revoke-keys" Oct 31 01:29:41.610376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.610331 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc39e123-28ad-4c99-9973-04ade69e2ae4" containerName="revoke-keys" Oct 31 01:29:41.610376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.610354 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f2323d-4169-4e62-a62d-a8a3e2d1f595" containerName="istio-proxy" Oct 31 01:29:41.613783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.613762 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.625421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.625398 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.625524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.625462 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.625620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.625602 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9p2m\" (UniqueName: \"kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.717316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.717279 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:29:41.718231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.718207 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:29:41.722843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.722807 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr"] Oct 31 01:29:41.726339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.726312 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.726462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.726380 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.726513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.726478 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9p2m\" (UniqueName: \"kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.727108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.727075 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.729030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.729003 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.783138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.783110 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9p2m\" (UniqueName: \"kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m\") pod \"maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:41.928338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:41.928256 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:42.309432 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:29:42.309400 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59202344_fb76_4a4d_989c_19b1e071a560.slice/crio-27a5adcc7317a23100a8f4ae7d1acaf282d10a5f7a79d44ab1b02d56f42b5688 WatchSource:0}: Error finding container 27a5adcc7317a23100a8f4ae7d1acaf282d10a5f7a79d44ab1b02d56f42b5688: Status 404 returned error can't find the container with id 27a5adcc7317a23100a8f4ae7d1acaf282d10a5f7a79d44ab1b02d56f42b5688 Oct 31 01:29:42.313183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:42.313154 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr"] Oct 31 01:29:42.739780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:42.739731 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" event={"ID":"59202344-fb76-4a4d-989c-19b1e071a560","Type":"ContainerStarted","Data":"9f5002c0d2736c51189ffbe19410ec9898465154405d32e1bf13e6b3a16c60b3"} Oct 31 01:29:42.740288 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:42.739796 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" event={"ID":"59202344-fb76-4a4d-989c-19b1e071a560","Type":"ContainerStarted","Data":"27a5adcc7317a23100a8f4ae7d1acaf282d10a5f7a79d44ab1b02d56f42b5688"} Oct 31 01:29:42.854227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:42.854176 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" podStartSLOduration=1.8541621529999999 podStartE2EDuration="1.854162153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:42.852609216 +0000 UTC m=+1954.659576760" watchObservedRunningTime="2026-10-31 01:29:42.854162153 +0000 UTC m=+1954.661129713" Oct 31 01:29:43.748112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:43.748078 5756 generic.go:356] "Generic (PLEG): container finished" podID="59202344-fb76-4a4d-989c-19b1e071a560" containerID="9f5002c0d2736c51189ffbe19410ec9898465154405d32e1bf13e6b3a16c60b3" exitCode=0 Oct 31 01:29:43.748513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:43.748172 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" event={"ID":"59202344-fb76-4a4d-989c-19b1e071a560","Type":"ContainerDied","Data":"9f5002c0d2736c51189ffbe19410ec9898465154405d32e1bf13e6b3a16c60b3"} Oct 31 01:29:44.907671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.907646 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:44.967734 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.967702 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m\" (UniqueName: \"kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m\") pod \"59202344-fb76-4a4d-989c-19b1e071a560\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " Oct 31 01:29:44.967945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.967913 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca\") pod \"59202344-fb76-4a4d-989c-19b1e071a560\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " Oct 31 01:29:44.968007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.967990 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca\") pod \"59202344-fb76-4a4d-989c-19b1e071a560\" (UID: \"59202344-fb76-4a4d-989c-19b1e071a560\") " Oct 31 01:29:44.968351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.968323 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca" pod "59202344-fb76-4a4d-989c-19b1e071a560" (UID: "59202344-fb76-4a4d-989c-19b1e071a560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:29:44.968615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.968563 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:44.970149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.970123 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca" pod "59202344-fb76-4a4d-989c-19b1e071a560" (UID: "59202344-fb76-4a4d-989c-19b1e071a560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:29:44.970149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:44.970133 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m" pod "59202344-fb76-4a4d-989c-19b1e071a560" (UID: "59202344-fb76-4a4d-989c-19b1e071a560"). InnerVolumeSpecName "kube-api-access-c9p2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:29:45.070166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:45.070136 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9p2m\" (UniqueName: \"kubernetes.io/projected/59202344-fb76-4a4d-989c-19b1e071a560-kube-api-access-c9p2m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:45.070166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:45.070162 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59202344-fb76-4a4d-989c-19b1e071a560-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:29:45.762650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:45.762621 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" Oct 31 01:29:45.762830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:45.762658 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr" event={"ID":"59202344-fb76-4a4d-989c-19b1e071a560","Type":"ContainerDied","Data":"27a5adcc7317a23100a8f4ae7d1acaf282d10a5f7a79d44ab1b02d56f42b5688"} Oct 31 01:29:45.762830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:45.762706 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27a5adcc7317a23100a8f4ae7d1acaf282d10a5f7a79d44ab1b02d56f42b5688" Oct 31 01:29:50.333773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:50.333739 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w"] Oct 31 01:29:50.383054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:50.383015 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f259852e-wcx7w"] Oct 31 01:29:52.311241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:52.311204 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc39e123-28ad-4c99-9973-04ade69e2ae4" path="/var/lib/kubelet/pods/cc39e123-28ad-4c99-9973-04ade69e2ae4/volumes" Oct 31 01:29:55.747757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:55.747722 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh"] Oct 31 01:29:55.817119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:29:55.817083 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:30:00.172507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.172477 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp"] Oct 31 01:30:00.173212 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.173195 5756 cpu_manager.go:414] "containerMap: removing container" podUID="59202344-fb76-4a4d-989c-19b1e071a560" containerName="revoke-keys" Oct 31 01:30:00.173212 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.173213 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59202344-fb76-4a4d-989c-19b1e071a560" containerName="revoke-keys" Oct 31 01:30:00.173379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.173328 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="59202344-fb76-4a4d-989c-19b1e071a560" containerName="revoke-keys" Oct 31 01:30:00.177592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.177562 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s"] Oct 31 01:30:00.177816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.177675 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.181477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.181455 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr"] Oct 31 01:30:00.181602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.181571 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.185201 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.185180 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.222091 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.222062 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86"] Oct 31 01:30:00.226206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.226184 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm"] Oct 31 01:30:00.226329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.226311 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.229755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.229736 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n"] Oct 31 01:30:00.229881 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.229846 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.233277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.233259 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6"] Oct 31 01:30:00.233414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.233399 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.235697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.235681 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:30:00.237034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.237019 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.271282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.271256 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c"] Oct 31 01:30:00.273285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.273263 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.273406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.273328 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhnh5\" (UniqueName: \"kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5\") pod \"maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.275146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.275131 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.297193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.297160 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86"] Oct 31 01:30:00.301069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.301039 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s"] Oct 31 01:30:00.318266 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.318232 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr"] Oct 31 01:30:00.318651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.318302 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp"] Oct 31 01:30:00.336891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.336841 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6"] Oct 31 01:30:00.342311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.342282 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c"] Oct 31 01:30:00.374946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.374912 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.375123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375040 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.375123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375096 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.375236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375159 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx66m\" (UniqueName: \"kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m\") pod \"maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.375598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375574 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.375741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375607 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5mhq\" (UniqueName: \"kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.375741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375667 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhnh5\" (UniqueName: \"kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5\") pod \"maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.375741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375692 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.375963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375805 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64m4\" (UniqueName: \"kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4\") pod \"maas-api-key-cleanup-29890170-flf6c\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.375963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.375841 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvl4\" (UniqueName: \"kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4\") pod \"maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.376115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376010 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.376115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376037 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrps\" (UniqueName: \"kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.376115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376091 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.376271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376139 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.376271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376170 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.376271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376189 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpr67\" (UniqueName: \"kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.376271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.376252 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-flf6c\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.396273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.396236 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm"] Oct 31 01:30:00.409565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.407644 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n"] Oct 31 01:30:00.433748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.433675 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhnh5\" (UniqueName: \"kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5\") pod \"maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.477650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.477622 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx66m\" (UniqueName: \"kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m\") pod \"maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.477834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.477655 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.477834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.477675 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5mhq\" (UniqueName: \"kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.477834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.477743 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.477834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.477794 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64m4\" (UniqueName: \"kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4\") pod \"maas-api-key-cleanup-29890170-flf6c\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.477834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.477830 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvl4\" (UniqueName: \"kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4\") pod \"maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.478135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478011 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrps\" (UniqueName: \"kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.478135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478036 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.478260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478122 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.478260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478171 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.478260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478202 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpr67\" (UniqueName: \"kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.478260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478244 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-flf6c\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.478475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478293 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.478475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478359 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.478644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478617 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.478806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478709 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.478936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478879 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.478936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.478897 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.479232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.479210 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.479274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.479258 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.479326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.479307 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890170-flf6c\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.494534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.494513 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:00.536066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.536023 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx66m\" (UniqueName: \"kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m\") pod \"maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.536066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.536061 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64m4\" (UniqueName: \"kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4\") pod \"maas-api-key-cleanup-29890170-flf6c\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.536292 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.536082 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvl4\" (UniqueName: \"kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4\") pod \"maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.536478 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.536462 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5mhq\" (UniqueName: \"kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.536833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.536817 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.537298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.537281 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrps\" (UniqueName: \"kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.537546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.537528 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpr67\" (UniqueName: \"kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.542580 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.542558 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:00.555920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.551382 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:00.560045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.559187 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:00.569081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.566530 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:00.593937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.593342 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:00.802790 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.802755 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:00.809147 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.809116 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:00.900915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.900883 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp"] Oct 31 01:30:00.901024 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:00.900556 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833600e3_cf61_4dfc_b2f5_00b2dfbe3de8.slice/crio-f446c5587b4d6bad4ae3612713427779b3f92262c2d404cad050ca8eac6b432a WatchSource:0}: Error finding container f446c5587b4d6bad4ae3612713427779b3f92262c2d404cad050ca8eac6b432a: Status 404 returned error can't find the container with id f446c5587b4d6bad4ae3612713427779b3f92262c2d404cad050ca8eac6b432a Oct 31 01:30:00.978361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.978279 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n"] Oct 31 01:30:00.979552 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:00.979522 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a557d7_c554_4596_a4be_ac7c7008ed8a.slice/crio-19ae2d09cbfc57ff6b5b3ba536cc221fa1fe30394abe2c2e90b88da83d050531 WatchSource:0}: Error finding container 19ae2d09cbfc57ff6b5b3ba536cc221fa1fe30394abe2c2e90b88da83d050531: Status 404 returned error can't find the container with id 19ae2d09cbfc57ff6b5b3ba536cc221fa1fe30394abe2c2e90b88da83d050531 Oct 31 01:30:00.996631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:00.996604 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86"] Oct 31 01:30:00.997356 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:00.997326 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7793d2_a85d_4c6f_814e_58ff1659b154.slice/crio-08007962d855dd3fc7476bb22e65addfaf7d863ffb35bdccede44fe8b499d858 WatchSource:0}: Error finding container 08007962d855dd3fc7476bb22e65addfaf7d863ffb35bdccede44fe8b499d858: Status 404 returned error can't find the container with id 08007962d855dd3fc7476bb22e65addfaf7d863ffb35bdccede44fe8b499d858 Oct 31 01:30:01.027235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.027212 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm"] Oct 31 01:30:01.031731 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:01.031701 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9d753f_74ee_4631_bb1d_6e51d584aed9.slice/crio-c3ead88c43a845a2703f9ec1f3a9f48f550262fec46f7ec80388f5d1b7004fda WatchSource:0}: Error finding container c3ead88c43a845a2703f9ec1f3a9f48f550262fec46f7ec80388f5d1b7004fda: Status 404 returned error can't find the container with id c3ead88c43a845a2703f9ec1f3a9f48f550262fec46f7ec80388f5d1b7004fda Oct 31 01:30:01.041429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.041408 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6"] Oct 31 01:30:01.044877 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:01.044829 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccac6f4e_4ee2_4c71_aeaf_cf8fdd7204ec.slice/crio-8d89d444137dfd9c97920dd8a173842135c60d9c5713c6fbe32d2c287d372146 WatchSource:0}: Error finding container 8d89d444137dfd9c97920dd8a173842135c60d9c5713c6fbe32d2c287d372146: Status 404 returned error can't find the container with id 8d89d444137dfd9c97920dd8a173842135c60d9c5713c6fbe32d2c287d372146 Oct 31 01:30:01.153230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.153209 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c"] Oct 31 01:30:01.154254 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:01.154231 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d4176d_9f4a_4841_911b_adeeee24c7db.slice/crio-e84e62958190fc94246a70501d69d3df3f54bcfcc9896c96cf91dd7db1153634 WatchSource:0}: Error finding container e84e62958190fc94246a70501d69d3df3f54bcfcc9896c96cf91dd7db1153634: Status 404 returned error can't find the container with id e84e62958190fc94246a70501d69d3df3f54bcfcc9896c96cf91dd7db1153634 Oct 31 01:30:01.213954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.213925 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s"] Oct 31 01:30:01.216914 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:01.216412 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3ea459_55d9_4c02_aaba_af8dd03f3406.slice/crio-7bf8b7ac43a32dabf3f6f7da3acbfbcdc7cf7d6412313357c7c8492de4d7a921 WatchSource:0}: Error finding container 7bf8b7ac43a32dabf3f6f7da3acbfbcdc7cf7d6412313357c7c8492de4d7a921: Status 404 returned error can't find the container with id 7bf8b7ac43a32dabf3f6f7da3acbfbcdc7cf7d6412313357c7c8492de4d7a921 Oct 31 01:30:01.217295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.217269 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr"] Oct 31 01:30:01.912651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.912615 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerStarted","Data":"c5c976716cd37643ca8004234f3ddbef7466054538b59a8d2786409250d1234e"} Oct 31 01:30:01.914367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.914340 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerStarted","Data":"f60c67be05ccd7224837dccd2caa7f37ea28959a2b653fc2c1f11d403c4d470c"} Oct 31 01:30:01.914521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.914387 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerStarted","Data":"f446c5587b4d6bad4ae3612713427779b3f92262c2d404cad050ca8eac6b432a"} Oct 31 01:30:01.915671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.915647 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerStarted","Data":"7bf8b7ac43a32dabf3f6f7da3acbfbcdc7cf7d6412313357c7c8492de4d7a921"} Oct 31 01:30:01.917214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.917190 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerStarted","Data":"f0bb55e8431c8038fe825004297e167d406c983451deb25198f7f72306171b81"} Oct 31 01:30:01.917320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.917236 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerStarted","Data":"19ae2d09cbfc57ff6b5b3ba536cc221fa1fe30394abe2c2e90b88da83d050531"} Oct 31 01:30:01.918650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.918629 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerStarted","Data":"96673e4149fa92cf86f5b5b1be26533e9d241f368df4ade21fd412acd8f9c080"} Oct 31 01:30:01.918746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.918670 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerStarted","Data":"8d89d444137dfd9c97920dd8a173842135c60d9c5713c6fbe32d2c287d372146"} Oct 31 01:30:01.920116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.920095 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerStarted","Data":"ce39d705a9e348f74a5bc483c5a6ffda8c15d3084b1d6b0c14491ecbf1f7aead"} Oct 31 01:30:01.920212 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.920133 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerStarted","Data":"e84e62958190fc94246a70501d69d3df3f54bcfcc9896c96cf91dd7db1153634"} Oct 31 01:30:01.921594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.921573 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" event={"ID":"9e9d753f-74ee-4631-bb1d-6e51d584aed9","Type":"ContainerStarted","Data":"7a7f2374f5435284010581dc18ddf215173edf491e8edad7eebadc8c76f2df69"} Oct 31 01:30:01.921657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.921616 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" event={"ID":"9e9d753f-74ee-4631-bb1d-6e51d584aed9","Type":"ContainerStarted","Data":"c3ead88c43a845a2703f9ec1f3a9f48f550262fec46f7ec80388f5d1b7004fda"} Oct 31 01:30:01.923081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.923059 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerStarted","Data":"534ac2cc4ff0223d7e127e0e7fd2d520ea6cd5af9d227a7a68ce994f7db8d637"} Oct 31 01:30:01.923165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:01.923103 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerStarted","Data":"08007962d855dd3fc7476bb22e65addfaf7d863ffb35bdccede44fe8b499d858"} Oct 31 01:30:02.027995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.027944 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" podStartSLOduration=2.027928392 podStartE2EDuration="2.027928392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.025758646 +0000 UTC m=+1973.832726190" watchObservedRunningTime="2026-10-31 01:30:02.027928392 +0000 UTC m=+1973.834895935" Oct 31 01:30:02.070191 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:02.070161 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:30:02.139853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.139800 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" podStartSLOduration=2.13978702 podStartE2EDuration="2.13978702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.138064914 +0000 UTC m=+1973.945032459" watchObservedRunningTime="2026-10-31 01:30:02.13978702 +0000 UTC m=+1973.946754560" Oct 31 01:30:02.251377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.251330 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" podStartSLOduration=2.251316902 podStartE2EDuration="2.251316902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.249378672 +0000 UTC m=+1974.056346215" watchObservedRunningTime="2026-10-31 01:30:02.251316902 +0000 UTC m=+1974.058284444" Oct 31 01:30:02.363546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.363501 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" podStartSLOduration=2.363487366 podStartE2EDuration="2.363487366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.362174372 +0000 UTC m=+1974.169141916" watchObservedRunningTime="2026-10-31 01:30:02.363487366 +0000 UTC m=+1974.170454910" Oct 31 01:30:02.477110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.477045 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" podStartSLOduration=2.477026421 podStartE2EDuration="2.477026421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.473581611 +0000 UTC m=+1974.280549155" watchObservedRunningTime="2026-10-31 01:30:02.477026421 +0000 UTC m=+1974.283993965" Oct 31 01:30:02.588792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.588737 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" podStartSLOduration=2.5887218389999997 podStartE2EDuration="2.588721839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:02.585122774 +0000 UTC m=+1974.392090319" watchObservedRunningTime="2026-10-31 01:30:02.588721839 +0000 UTC m=+1974.395689381" Oct 31 01:30:02.930930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.930897 5756 generic.go:356] "Generic (PLEG): container finished" podID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerID="96673e4149fa92cf86f5b5b1be26533e9d241f368df4ade21fd412acd8f9c080" exitCode=60 Oct 31 01:30:02.931114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.930964 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerDied","Data":"96673e4149fa92cf86f5b5b1be26533e9d241f368df4ade21fd412acd8f9c080"} Oct 31 01:30:02.931271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.931253 5756 scope.go:122] "RemoveContainer" containerID="96673e4149fa92cf86f5b5b1be26533e9d241f368df4ade21fd412acd8f9c080" Oct 31 01:30:02.932749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.932717 5756 generic.go:356] "Generic (PLEG): container finished" podID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerID="ce39d705a9e348f74a5bc483c5a6ffda8c15d3084b1d6b0c14491ecbf1f7aead" exitCode=60 Oct 31 01:30:02.932849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.932784 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerDied","Data":"ce39d705a9e348f74a5bc483c5a6ffda8c15d3084b1d6b0c14491ecbf1f7aead"} Oct 31 01:30:02.933063 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.933048 5756 scope.go:122] "RemoveContainer" containerID="ce39d705a9e348f74a5bc483c5a6ffda8c15d3084b1d6b0c14491ecbf1f7aead" Oct 31 01:30:02.934799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.934776 5756 generic.go:356] "Generic (PLEG): container finished" podID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerID="7a7f2374f5435284010581dc18ddf215173edf491e8edad7eebadc8c76f2df69" exitCode=60 Oct 31 01:30:02.934919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.934880 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" event={"ID":"9e9d753f-74ee-4631-bb1d-6e51d584aed9","Type":"ContainerDied","Data":"7a7f2374f5435284010581dc18ddf215173edf491e8edad7eebadc8c76f2df69"} Oct 31 01:30:02.935117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.935100 5756 scope.go:122] "RemoveContainer" containerID="7a7f2374f5435284010581dc18ddf215173edf491e8edad7eebadc8c76f2df69" Oct 31 01:30:02.936767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.936747 5756 generic.go:356] "Generic (PLEG): container finished" podID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerID="534ac2cc4ff0223d7e127e0e7fd2d520ea6cd5af9d227a7a68ce994f7db8d637" exitCode=60 Oct 31 01:30:02.936885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.936822 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerDied","Data":"534ac2cc4ff0223d7e127e0e7fd2d520ea6cd5af9d227a7a68ce994f7db8d637"} Oct 31 01:30:02.937063 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.937046 5756 scope.go:122] "RemoveContainer" containerID="534ac2cc4ff0223d7e127e0e7fd2d520ea6cd5af9d227a7a68ce994f7db8d637" Oct 31 01:30:02.938368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.938352 5756 generic.go:356] "Generic (PLEG): container finished" podID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerID="a4bfe15f353cf044ff759f4d7e7eb51125133520435c7ee89b3800ba25283187" exitCode=60 Oct 31 01:30:02.938472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.938436 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerDied","Data":"a4bfe15f353cf044ff759f4d7e7eb51125133520435c7ee89b3800ba25283187"} Oct 31 01:30:02.938598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.938586 5756 scope.go:122] "RemoveContainer" containerID="a4bfe15f353cf044ff759f4d7e7eb51125133520435c7ee89b3800ba25283187" Oct 31 01:30:02.940000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.939983 5756 generic.go:356] "Generic (PLEG): container finished" podID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerID="f60c67be05ccd7224837dccd2caa7f37ea28959a2b653fc2c1f11d403c4d470c" exitCode=60 Oct 31 01:30:02.940088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.940069 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerDied","Data":"f60c67be05ccd7224837dccd2caa7f37ea28959a2b653fc2c1f11d403c4d470c"} Oct 31 01:30:02.940226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.940213 5756 scope.go:122] "RemoveContainer" containerID="f60c67be05ccd7224837dccd2caa7f37ea28959a2b653fc2c1f11d403c4d470c" Oct 31 01:30:02.941520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.941494 5756 generic.go:356] "Generic (PLEG): container finished" podID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerID="60dc54c9e8568ca74652f09254e9b5755d50a05a8405d5bfe4e55b9a704c64ea" exitCode=60 Oct 31 01:30:02.941856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.941838 5756 scope.go:122] "RemoveContainer" containerID="60dc54c9e8568ca74652f09254e9b5755d50a05a8405d5bfe4e55b9a704c64ea" Oct 31 01:30:02.942097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.942076 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerDied","Data":"60dc54c9e8568ca74652f09254e9b5755d50a05a8405d5bfe4e55b9a704c64ea"} Oct 31 01:30:02.944393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.944373 5756 generic.go:356] "Generic (PLEG): container finished" podID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerID="f0bb55e8431c8038fe825004297e167d406c983451deb25198f7f72306171b81" exitCode=60 Oct 31 01:30:02.944488 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.944427 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerDied","Data":"f0bb55e8431c8038fe825004297e167d406c983451deb25198f7f72306171b81"} Oct 31 01:30:02.944653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:02.944638 5756 scope.go:122] "RemoveContainer" containerID="f0bb55e8431c8038fe825004297e167d406c983451deb25198f7f72306171b81" Oct 31 01:30:03.956641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.956604 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerStarted","Data":"75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1"} Oct 31 01:30:03.958853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.958824 5756 generic.go:356] "Generic (PLEG): container finished" podID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerID="8cc849ea5d7d6c281bdea2a18ac7a84f016292264c3afa8b60a414ab1c732498" exitCode=60 Oct 31 01:30:03.959015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.958894 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerDied","Data":"8cc849ea5d7d6c281bdea2a18ac7a84f016292264c3afa8b60a414ab1c732498"} Oct 31 01:30:03.959015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.958943 5756 scope.go:122] "RemoveContainer" containerID="f60c67be05ccd7224837dccd2caa7f37ea28959a2b653fc2c1f11d403c4d470c" Oct 31 01:30:03.959340 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.959320 5756 scope.go:122] "RemoveContainer" containerID="8cc849ea5d7d6c281bdea2a18ac7a84f016292264c3afa8b60a414ab1c732498" Oct 31 01:30:03.959719 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:03.959689 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp_odh-ai-gateway-infra(833600e3-cf61-4dfc-b2f5-00b2dfbe3de8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" Oct 31 01:30:03.961835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.961564 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerStarted","Data":"6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18"} Oct 31 01:30:03.963721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.963700 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerStarted","Data":"da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b"} Oct 31 01:30:03.965598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.965575 5756 generic.go:356] "Generic (PLEG): container finished" podID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerID="824940b0c4949a57bd3c96ad1cb5d0d408962c2500080ba30712692e201ce1b4" exitCode=60 Oct 31 01:30:03.965711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.965651 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerDied","Data":"824940b0c4949a57bd3c96ad1cb5d0d408962c2500080ba30712692e201ce1b4"} Oct 31 01:30:03.965939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.965924 5756 scope.go:122] "RemoveContainer" containerID="824940b0c4949a57bd3c96ad1cb5d0d408962c2500080ba30712692e201ce1b4" Oct 31 01:30:03.966194 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:03.966175 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6_odh-ai-gateway-infra(ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" Oct 31 01:30:03.967816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.967795 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerStarted","Data":"826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7"} Oct 31 01:30:03.969648 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.969627 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" event={"ID":"9e9d753f-74ee-4631-bb1d-6e51d584aed9","Type":"ContainerStarted","Data":"8723fdf4be79ed87b8c97f213d7f3a44b1bb4917dc5a7907d1405f72d4d6af62"} Oct 31 01:30:03.976319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.976283 5756 generic.go:356] "Generic (PLEG): container finished" podID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerID="a5e87bf6fabbdc3bde9bfc3e7e565b4168d0fa455d5c35f1ef4c081f7f62cb7f" exitCode=60 Oct 31 01:30:03.976410 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.976357 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerDied","Data":"a5e87bf6fabbdc3bde9bfc3e7e565b4168d0fa455d5c35f1ef4c081f7f62cb7f"} Oct 31 01:30:03.976612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:03.976597 5756 scope.go:122] "RemoveContainer" containerID="a5e87bf6fabbdc3bde9bfc3e7e565b4168d0fa455d5c35f1ef4c081f7f62cb7f" Oct 31 01:30:03.976915 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:03.976894 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86_odh-ai-gateway-infra(bf7793d2-a85d-4c6f-814e-58ff1659b154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" Oct 31 01:30:04.197920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.197854 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" podStartSLOduration=4.197840694 podStartE2EDuration="4.197840694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:04.196678214 +0000 UTC m=+1976.003645757" watchObservedRunningTime="2026-10-31 01:30:04.197840694 +0000 UTC m=+1976.004808237" Oct 31 01:30:04.199012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.198980 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" podStartSLOduration=4.198970665 podStartE2EDuration="4.198970665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:04.078541595 +0000 UTC m=+1975.885509138" watchObservedRunningTime="2026-10-31 01:30:04.198970665 +0000 UTC m=+1976.005938209" Oct 31 01:30:04.372053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.372027 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm"] Oct 31 01:30:04.417513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.417484 5756 scope.go:122] "RemoveContainer" containerID="96673e4149fa92cf86f5b5b1be26533e9d241f368df4ade21fd412acd8f9c080" Oct 31 01:30:04.436148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.436120 5756 scope.go:122] "RemoveContainer" containerID="534ac2cc4ff0223d7e127e0e7fd2d520ea6cd5af9d227a7a68ce994f7db8d637" Oct 31 01:30:04.985041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.985013 5756 scope.go:122] "RemoveContainer" containerID="8cc849ea5d7d6c281bdea2a18ac7a84f016292264c3afa8b60a414ab1c732498" Oct 31 01:30:04.985456 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.985377 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp_odh-ai-gateway-infra(833600e3-cf61-4dfc-b2f5-00b2dfbe3de8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" Oct 31 01:30:04.986776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.986756 5756 generic.go:356] "Generic (PLEG): container finished" podID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerID="6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18" exitCode=60 Oct 31 01:30:04.986875 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.986840 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerDied","Data":"6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18"} Oct 31 01:30:04.986950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.986935 5756 scope.go:122] "RemoveContainer" containerID="60dc54c9e8568ca74652f09254e9b5755d50a05a8405d5bfe4e55b9a704c64ea" Oct 31 01:30:04.987110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.987096 5756 scope.go:122] "RemoveContainer" containerID="6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18" Oct 31 01:30:04.987391 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.987369 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s_odh-ai-gateway-infra(0e3ea459-55d9-4c02-aaba-af8dd03f3406)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" Oct 31 01:30:04.988723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.988703 5756 generic.go:356] "Generic (PLEG): container finished" podID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerID="da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b" exitCode=60 Oct 31 01:30:04.988828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.988782 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerDied","Data":"da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b"} Oct 31 01:30:04.988972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.988953 5756 scope.go:122] "RemoveContainer" containerID="da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b" Oct 31 01:30:04.989204 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.989184 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n_odh-ai-gateway-infra(68a557d7-c554-4596-a4be-ac7c7008ed8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" Oct 31 01:30:04.990646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.990629 5756 scope.go:122] "RemoveContainer" containerID="824940b0c4949a57bd3c96ad1cb5d0d408962c2500080ba30712692e201ce1b4" Oct 31 01:30:04.990852 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.990826 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6_odh-ai-gateway-infra(ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" Oct 31 01:30:04.992594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.992569 5756 generic.go:356] "Generic (PLEG): container finished" podID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerID="826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7" exitCode=60 Oct 31 01:30:04.992674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.992646 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerDied","Data":"826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7"} Oct 31 01:30:04.992928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.992908 5756 scope.go:122] "RemoveContainer" containerID="826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7" Oct 31 01:30:04.993136 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.993116 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-flf6c_odh-ai-gateway-infra(e8d4176d-9f4a-4841-911b-adeeee24c7db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" Oct 31 01:30:04.994950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.994930 5756 generic.go:356] "Generic (PLEG): container finished" podID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerID="8723fdf4be79ed87b8c97f213d7f3a44b1bb4917dc5a7907d1405f72d4d6af62" exitCode=60 Oct 31 01:30:04.995035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.995012 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" event={"ID":"9e9d753f-74ee-4631-bb1d-6e51d584aed9","Type":"ContainerDied","Data":"8723fdf4be79ed87b8c97f213d7f3a44b1bb4917dc5a7907d1405f72d4d6af62"} Oct 31 01:30:04.995135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.995114 5756 scope.go:122] "RemoveContainer" containerID="8723fdf4be79ed87b8c97f213d7f3a44b1bb4917dc5a7907d1405f72d4d6af62" Oct 31 01:30:04.995374 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.995351 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm_odh-ai-gateway-infra(9e9d753f-74ee-4631-bb1d-6e51d584aed9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" Oct 31 01:30:04.996934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.996915 5756 scope.go:122] "RemoveContainer" containerID="a5e87bf6fabbdc3bde9bfc3e7e565b4168d0fa455d5c35f1ef4c081f7f62cb7f" Oct 31 01:30:04.997161 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.997137 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86_odh-ai-gateway-infra(bf7793d2-a85d-4c6f-814e-58ff1659b154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" Oct 31 01:30:04.998612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.998592 5756 generic.go:356] "Generic (PLEG): container finished" podID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerID="75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1" exitCode=60 Oct 31 01:30:04.998699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.998629 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerDied","Data":"75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1"} Oct 31 01:30:04.998828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:04.998815 5756 scope.go:122] "RemoveContainer" containerID="75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1" Oct 31 01:30:04.999018 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:04.999002 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr_odh-ai-gateway-infra(9a97b71f-f33b-45ad-8ce8-093b120d13a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" Oct 31 01:30:05.001986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:05.001969 5756 scope.go:122] "RemoveContainer" containerID="f0bb55e8431c8038fe825004297e167d406c983451deb25198f7f72306171b81" Oct 31 01:30:05.014621 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:05.014602 5756 scope.go:122] "RemoveContainer" containerID="ce39d705a9e348f74a5bc483c5a6ffda8c15d3084b1d6b0c14491ecbf1f7aead" Oct 31 01:30:05.027132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:05.027110 5756 scope.go:122] "RemoveContainer" containerID="7a7f2374f5435284010581dc18ddf215173edf491e8edad7eebadc8c76f2df69" Oct 31 01:30:05.040671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:05.040653 5756 scope.go:122] "RemoveContainer" containerID="a4bfe15f353cf044ff759f4d7e7eb51125133520435c7ee89b3800ba25283187" Oct 31 01:30:06.012059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.012026 5756 scope.go:122] "RemoveContainer" containerID="826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7" Oct 31 01:30:06.012548 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:06.012406 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-flf6c_odh-ai-gateway-infra(e8d4176d-9f4a-4841-911b-adeeee24c7db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" Oct 31 01:30:06.017977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.017946 5756 scope.go:122] "RemoveContainer" containerID="75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1" Oct 31 01:30:06.018306 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:06.018275 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr_odh-ai-gateway-infra(9a97b71f-f33b-45ad-8ce8-093b120d13a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" Oct 31 01:30:06.023030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.023011 5756 scope.go:122] "RemoveContainer" containerID="6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18" Oct 31 01:30:06.023615 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:06.023512 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s_odh-ai-gateway-infra(0e3ea459-55d9-4c02-aaba-af8dd03f3406)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" Oct 31 01:30:06.027263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.027242 5756 scope.go:122] "RemoveContainer" containerID="da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b" Oct 31 01:30:06.027552 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:06.027524 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n_odh-ai-gateway-infra(68a557d7-c554-4596-a4be-ac7c7008ed8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" Oct 31 01:30:06.253212 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.253183 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:06.295379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.295349 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4\" (UniqueName: \"kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4\") pod \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " Oct 31 01:30:06.295531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.295396 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca\") pod \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\" (UID: \"9e9d753f-74ee-4631-bb1d-6e51d584aed9\") " Oct 31 01:30:06.295772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.295749 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca" pod "9e9d753f-74ee-4631-bb1d-6e51d584aed9" (UID: "9e9d753f-74ee-4631-bb1d-6e51d584aed9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:06.297617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.297594 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4" pod "9e9d753f-74ee-4631-bb1d-6e51d584aed9" (UID: "9e9d753f-74ee-4631-bb1d-6e51d584aed9"). InnerVolumeSpecName "kube-api-access-nlvl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:06.345663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.345633 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np"] Oct 31 01:30:06.348459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.348437 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f57e85-rr8np"] Oct 31 01:30:06.396798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.396771 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvl4\" (UniqueName: \"kubernetes.io/projected/9e9d753f-74ee-4631-bb1d-6e51d584aed9-kube-api-access-nlvl4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:06.396798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:06.396795 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e9d753f-74ee-4631-bb1d-6e51d584aed9-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:07.036727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:07.036697 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" Oct 31 01:30:07.037208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:07.036727 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm" event={"ID":"9e9d753f-74ee-4631-bb1d-6e51d584aed9","Type":"ContainerDied","Data":"c3ead88c43a845a2703f9ec1f3a9f48f550262fec46f7ec80388f5d1b7004fda"} Oct 31 01:30:07.037208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:07.036787 5756 scope.go:122] "RemoveContainer" containerID="8723fdf4be79ed87b8c97f213d7f3a44b1bb4917dc5a7907d1405f72d4d6af62" Oct 31 01:30:07.201055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:07.201023 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm"] Oct 31 01:30:07.203572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:07.203553 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-f57e85-29890170-plbkm"] Oct 31 01:30:08.312694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:08.312657 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af2ac60-f40f-46d1-ad21-b98a0b080979" path="/var/lib/kubelet/pods/0af2ac60-f40f-46d1-ad21-b98a0b080979/volumes" Oct 31 01:30:08.313318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:08.313296 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" path="/var/lib/kubelet/pods/9e9d753f-74ee-4631-bb1d-6e51d584aed9/volumes" Oct 31 01:30:09.535758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:09.535718 5756 scope.go:122] "RemoveContainer" containerID="879c174794f3fe3f3a7794876836d706de0f69ab59db92542ef0a497db32fe3c" Oct 31 01:30:10.769065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:10.769003 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" podUID="e0382435-c90b-452c-a93f-30d2741c65d8" containerName="main" containerID="cri-o://a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a" gracePeriod=45 Oct 31 01:30:11.053916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.053891 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:30:11.063689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.063659 5756 generic.go:356] "Generic (PLEG): container finished" podID="e0382435-c90b-452c-a93f-30d2741c65d8" containerID="a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a" exitCode=0 Oct 31 01:30:11.063852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.063712 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" Oct 31 01:30:11.063852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.063737 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" event={"ID":"e0382435-c90b-452c-a93f-30d2741c65d8","Type":"ContainerDied","Data":"a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a"} Oct 31 01:30:11.063852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.063783 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh" event={"ID":"e0382435-c90b-452c-a93f-30d2741c65d8","Type":"ContainerDied","Data":"affbfecf46572f7bebd4fc23f12f79764e535759159567cace79c238164a691f"} Oct 31 01:30:11.063852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.063801 5756 scope.go:122] "RemoveContainer" containerID="a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a" Oct 31 01:30:11.073741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.073719 5756 scope.go:122] "RemoveContainer" containerID="a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a" Oct 31 01:30:11.074035 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:11.074015 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a\": container with ID starting with a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a not found: ID does not exist" containerID="a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a" Oct 31 01:30:11.074100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.074044 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a"} err="failed to get container status \"a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a\": rpc error: code = NotFound desc = could not find container \"a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a\": container with ID starting with a0345954b808321b32c2ecb19273dd623457573e57f32de73bb7d7fc6821ac6a not found: ID does not exist" Oct 31 01:30:11.156141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156101 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm\") pod \"e0382435-c90b-452c-a93f-30d2741c65d8\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " Oct 31 01:30:11.156293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156210 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home\") pod \"e0382435-c90b-452c-a93f-30d2741c65d8\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " Oct 31 01:30:11.156330 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156297 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache\") pod \"e0382435-c90b-452c-a93f-30d2741c65d8\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " Oct 31 01:30:11.156330 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156326 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir\") pod \"e0382435-c90b-452c-a93f-30d2741c65d8\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " Oct 31 01:30:11.156433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156419 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7\" (UniqueName: \"kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7\") pod \"e0382435-c90b-452c-a93f-30d2741c65d8\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " Oct 31 01:30:11.156497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156452 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs\") pod \"e0382435-c90b-452c-a93f-30d2741c65d8\" (UID: \"e0382435-c90b-452c-a93f-30d2741c65d8\") " Oct 31 01:30:11.156551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156496 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home" pod "e0382435-c90b-452c-a93f-30d2741c65d8" (UID: "e0382435-c90b-452c-a93f-30d2741c65d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:11.156551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156510 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache" pod "e0382435-c90b-452c-a93f-30d2741c65d8" (UID: "e0382435-c90b-452c-a93f-30d2741c65d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:11.156687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.156665 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir" pod "e0382435-c90b-452c-a93f-30d2741c65d8" (UID: "e0382435-c90b-452c-a93f-30d2741c65d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:11.157048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.157026 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:11.157151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.157056 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:11.157151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.157071 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:11.158572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.158548 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm" pod "e0382435-c90b-452c-a93f-30d2741c65d8" (UID: "e0382435-c90b-452c-a93f-30d2741c65d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:11.158661 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.158640 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs" pod "e0382435-c90b-452c-a93f-30d2741c65d8" (UID: "e0382435-c90b-452c-a93f-30d2741c65d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:11.158725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.158673 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7" pod "e0382435-c90b-452c-a93f-30d2741c65d8" (UID: "e0382435-c90b-452c-a93f-30d2741c65d8"). InnerVolumeSpecName "kube-api-access-66vr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:11.258451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.258422 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66vr7\" (UniqueName: \"kubernetes.io/projected/e0382435-c90b-452c-a93f-30d2741c65d8-kube-api-access-66vr7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:11.258451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.258450 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0382435-c90b-452c-a93f-30d2741c65d8-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:11.258451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.258460 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0382435-c90b-452c-a93f-30d2741c65d8-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:11.483820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.483790 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh"] Oct 31 01:30:11.487715 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:11.487687 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f259852e/e2e-lifecycle-model-f259852e-kserve-687d6f7b9c-5q7bh"] Oct 31 01:30:12.312312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:12.312277 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0382435-c90b-452c-a93f-30d2741c65d8" path="/var/lib/kubelet/pods/e0382435-c90b-452c-a93f-30d2741c65d8/volumes" Oct 31 01:30:12.858665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:12.858626 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd"] Oct 31 01:30:12.858905 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:12.858840 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" containerID="cri-o://7f95f303608648880bffc55e68d30c2c0d6f807617b51d9e8f15754d5cf623fd" gracePeriod=30 Oct 31 01:30:15.304798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:15.304771 5756 scope.go:122] "RemoveContainer" containerID="826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7" Oct 31 01:30:15.305252 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:15.304918 5756 scope.go:122] "RemoveContainer" containerID="824940b0c4949a57bd3c96ad1cb5d0d408962c2500080ba30712692e201ce1b4" Oct 31 01:30:16.101720 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:16.101681 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerStarted","Data":"55a08e5e1c91b76d3a93cef38020d43c4daedef10e8df9d6ba1891ef70ebe589"} Oct 31 01:30:16.103814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:16.103787 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerStarted","Data":"ac4b12705a6ac6a147f0b41de5dfcdc6e3a3ee13466eafbbea4f838053910916"} Oct 31 01:30:16.304339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:16.304311 5756 scope.go:122] "RemoveContainer" containerID="da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b" Oct 31 01:30:16.304498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:16.304450 5756 scope.go:122] "RemoveContainer" containerID="6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18" Oct 31 01:30:17.112029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.111986 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerStarted","Data":"5df3f42428ea373ea5f967662ff3eb6efcc0f8667f477da75f2f07ee2cbfd665"} Oct 31 01:30:17.114074 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.114046 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerStarted","Data":"7e371c89f46a3311bd46e4433012d42f26332a9735bfb7fa9625d44943067d67"} Oct 31 01:30:17.115948 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.115915 5756 generic.go:356] "Generic (PLEG): container finished" podID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerID="55a08e5e1c91b76d3a93cef38020d43c4daedef10e8df9d6ba1891ef70ebe589" exitCode=60 Oct 31 01:30:17.115948 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.115939 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerDied","Data":"55a08e5e1c91b76d3a93cef38020d43c4daedef10e8df9d6ba1891ef70ebe589"} Oct 31 01:30:17.116085 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.115984 5756 scope.go:122] "RemoveContainer" containerID="824940b0c4949a57bd3c96ad1cb5d0d408962c2500080ba30712692e201ce1b4" Oct 31 01:30:17.116320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.116306 5756 scope.go:122] "RemoveContainer" containerID="55a08e5e1c91b76d3a93cef38020d43c4daedef10e8df9d6ba1891ef70ebe589" Oct 31 01:30:17.116635 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:17.116616 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6_odh-ai-gateway-infra(ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" Oct 31 01:30:17.117950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.117925 5756 generic.go:356] "Generic (PLEG): container finished" podID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerID="ac4b12705a6ac6a147f0b41de5dfcdc6e3a3ee13466eafbbea4f838053910916" exitCode=60 Oct 31 01:30:17.118035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.117981 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerDied","Data":"ac4b12705a6ac6a147f0b41de5dfcdc6e3a3ee13466eafbbea4f838053910916"} Oct 31 01:30:17.118307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.118288 5756 scope.go:122] "RemoveContainer" containerID="ac4b12705a6ac6a147f0b41de5dfcdc6e3a3ee13466eafbbea4f838053910916" Oct 31 01:30:17.118557 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:17.118540 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890170-flf6c_odh-ai-gateway-infra(e8d4176d-9f4a-4841-911b-adeeee24c7db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" Oct 31 01:30:17.128815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.128784 5756 scope.go:122] "RemoveContainer" containerID="826d5b68fd6900d3027a0095b03c5f089206a1a135f2c6c57f3e3c60818eb0a7" Oct 31 01:30:17.229178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.229146 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6"] Oct 31 01:30:17.346347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.346317 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c"] Oct 31 01:30:17.653594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.653559 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf"] Oct 31 01:30:17.661987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:17.661954 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-88d9b8-b9zjf"] Oct 31 01:30:18.126687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.126616 5756 generic.go:356] "Generic (PLEG): container finished" podID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerID="7f95f303608648880bffc55e68d30c2c0d6f807617b51d9e8f15754d5cf623fd" exitCode=0 Oct 31 01:30:18.127129 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.126682 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" event={"ID":"10abdb0d-7e36-4d79-bd39-4f82b09560df","Type":"ContainerDied","Data":"7f95f303608648880bffc55e68d30c2c0d6f807617b51d9e8f15754d5cf623fd"} Oct 31 01:30:18.127129 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.126757 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" event={"ID":"10abdb0d-7e36-4d79-bd39-4f82b09560df","Type":"ContainerDied","Data":"8f2c52aa17f304a28f0cc152193a22b13cd5be3218c654b7f826f9a646fce31a"} Oct 31 01:30:18.127129 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.126787 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2c52aa17f304a28f0cc152193a22b13cd5be3218c654b7f826f9a646fce31a" Oct 31 01:30:18.127129 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.126805 5756 scope.go:122] "RemoveContainer" containerID="ac12fafacc6fc495c3a9c7cd4f04bb1f3940a55cd6d5195c329fe42c8f1c0786" Oct 31 01:30:18.133174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.133147 5756 generic.go:356] "Generic (PLEG): container finished" podID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerID="5df3f42428ea373ea5f967662ff3eb6efcc0f8667f477da75f2f07ee2cbfd665" exitCode=60 Oct 31 01:30:18.133285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.133193 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerDied","Data":"5df3f42428ea373ea5f967662ff3eb6efcc0f8667f477da75f2f07ee2cbfd665"} Oct 31 01:30:18.133649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.133633 5756 scope.go:122] "RemoveContainer" containerID="5df3f42428ea373ea5f967662ff3eb6efcc0f8667f477da75f2f07ee2cbfd665" Oct 31 01:30:18.134058 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:18.134028 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s_odh-ai-gateway-infra(0e3ea459-55d9-4c02-aaba-af8dd03f3406)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" Oct 31 01:30:18.135450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.135427 5756 generic.go:356] "Generic (PLEG): container finished" podID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerID="7e371c89f46a3311bd46e4433012d42f26332a9735bfb7fa9625d44943067d67" exitCode=60 Oct 31 01:30:18.135538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.135486 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerDied","Data":"7e371c89f46a3311bd46e4433012d42f26332a9735bfb7fa9625d44943067d67"} Oct 31 01:30:18.135930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.135845 5756 scope.go:122] "RemoveContainer" containerID="7e371c89f46a3311bd46e4433012d42f26332a9735bfb7fa9625d44943067d67" Oct 31 01:30:18.136228 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:18.136203 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n_odh-ai-gateway-infra(68a557d7-c554-4596-a4be-ac7c7008ed8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" Oct 31 01:30:18.145042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.144167 5756 scope.go:122] "RemoveContainer" containerID="6182378cf0254acdf4c9c2707e0518db42ef657b36c34a30c9bbdd40e2414d18" Oct 31 01:30:18.148727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.148703 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:30:18.161198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.161176 5756 scope.go:122] "RemoveContainer" containerID="da4bdd0b060553a992f66c2a21ebbfce2b5dd0c5fbe438fd85c1933e4ee26c1b" Oct 31 01:30:18.238790 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.238757 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s"] Oct 31 01:30:18.257697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.257665 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.257905 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.257880 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.257921 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.257964 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258021 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258108 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258132 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:18.258215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258157 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258226 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258259 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket\") pod \"10abdb0d-7e36-4d79-bd39-4f82b09560df\" (UID: \"10abdb0d-7e36-4d79-bd39-4f82b09560df\") " Oct 31 01:30:18.258378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258328 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:18.259144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258957 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.259144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.258982 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.259144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.259035 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:18.259797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.259423 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:18.260347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.259946 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:18.264141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.264081 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:30:18.271504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.270094 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "kube-api-access-mwjtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:18.273311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.273275 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:18.273795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.273761 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token" pod "10abdb0d-7e36-4d79-bd39-4f82b09560df" (UID: "10abdb0d-7e36-4d79-bd39-4f82b09560df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:18.288503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.288478 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:18.304334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.304312 5756 scope.go:122] "RemoveContainer" containerID="8cc849ea5d7d6c281bdea2a18ac7a84f016292264c3afa8b60a414ab1c732498" Oct 31 01:30:18.306313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.304539 5756 scope.go:122] "RemoveContainer" containerID="a5e87bf6fabbdc3bde9bfc3e7e565b4168d0fa455d5c35f1ef4c081f7f62cb7f" Oct 31 01:30:18.306313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.304645 5756 scope.go:122] "RemoveContainer" containerID="75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1" Oct 31 01:30:18.313468 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.313437 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6cd153c-292b-4404-b7fe-38b44a59648b" path="/var/lib/kubelet/pods/c6cd153c-292b-4404-b7fe-38b44a59648b/volumes" Oct 31 01:30:18.325601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.325583 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:18.351473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.351448 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n"] Oct 31 01:30:18.359858 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.359835 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca\") pod \"e8d4176d-9f4a-4841-911b-adeeee24c7db\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " Oct 31 01:30:18.360052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.360037 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4\" (UniqueName: \"kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4\") pod \"e8d4176d-9f4a-4841-911b-adeeee24c7db\" (UID: \"e8d4176d-9f4a-4841-911b-adeeee24c7db\") " Oct 31 01:30:18.360263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.360242 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca" pod "e8d4176d-9f4a-4841-911b-adeeee24c7db" (UID: "e8d4176d-9f4a-4841-911b-adeeee24c7db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:18.361174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361150 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjtc\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-kube-api-access-mwjtc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361182 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361199 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d4176d-9f4a-4841-911b-adeeee24c7db-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361214 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361231 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361245 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361257 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abdb0d-7e36-4d79-bd39-4f82b09560df-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.361443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.361270 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abdb0d-7e36-4d79-bd39-4f82b09560df-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.362348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.362330 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4" pod "e8d4176d-9f4a-4841-911b-adeeee24c7db" (UID: "e8d4176d-9f4a-4841-911b-adeeee24c7db"). InnerVolumeSpecName "kube-api-access-h64m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:18.462318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.462287 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq\" (UniqueName: \"kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq\") pod \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " Oct 31 01:30:18.462443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.462389 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca\") pod \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\" (UID: \"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec\") " Oct 31 01:30:18.462785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.462760 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca" pod "ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" (UID: "ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:18.463082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.462830 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64m4\" (UniqueName: \"kubernetes.io/projected/e8d4176d-9f4a-4841-911b-adeeee24c7db-kube-api-access-h64m4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.529554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.529522 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq" pod "ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" (UID: "ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec"). InnerVolumeSpecName "kube-api-access-v5mhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:18.564036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.564011 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5mhq\" (UniqueName: \"kubernetes.io/projected/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-kube-api-access-v5mhq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:18.564036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:18.564036 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.143424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.143399 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" Oct 31 01:30:19.143424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.143406 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6" event={"ID":"ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec","Type":"ContainerDied","Data":"8d89d444137dfd9c97920dd8a173842135c60d9c5713c6fbe32d2c287d372146"} Oct 31 01:30:19.143945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.143508 5756 scope.go:122] "RemoveContainer" containerID="55a08e5e1c91b76d3a93cef38020d43c4daedef10e8df9d6ba1891ef70ebe589" Oct 31 01:30:19.144891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.144875 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" Oct 31 01:30:19.144968 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.144898 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c" event={"ID":"e8d4176d-9f4a-4841-911b-adeeee24c7db","Type":"ContainerDied","Data":"e84e62958190fc94246a70501d69d3df3f54bcfcc9896c96cf91dd7db1153634"} Oct 31 01:30:19.147032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.147004 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerStarted","Data":"50a909f28ec2c412ddfcf0002313f473312340257a5262707b2a6c6f1237f4cb"} Oct 31 01:30:19.149241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.149215 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerStarted","Data":"8456f8195d0727f6ccfcba9493b2e121c14877cb9a69a316b22a77a6efe51e02"} Oct 31 01:30:19.151330 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.151304 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerStarted","Data":"f607e046f17a7c78f7587bf44972dda05007d5978bb864819954a4a1d0fe6282"} Oct 31 01:30:19.156917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.156898 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd" Oct 31 01:30:19.157017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.156921 5756 scope.go:122] "RemoveContainer" containerID="ac4b12705a6ac6a147f0b41de5dfcdc6e3a3ee13466eafbbea4f838053910916" Oct 31 01:30:19.428338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.428267 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:19.531370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.531340 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6"] Oct 31 01:30:19.533792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.533772 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890170-fx2v6"] Oct 31 01:30:19.578132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.578105 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps\" (UniqueName: \"kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps\") pod \"68a557d7-c554-4596-a4be-ac7c7008ed8a\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " Oct 31 01:30:19.578295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.578266 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca\") pod \"68a557d7-c554-4596-a4be-ac7c7008ed8a\" (UID: \"68a557d7-c554-4596-a4be-ac7c7008ed8a\") " Oct 31 01:30:19.578665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.578641 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca" pod "68a557d7-c554-4596-a4be-ac7c7008ed8a" (UID: "68a557d7-c554-4596-a4be-ac7c7008ed8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:19.578795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.578781 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a557d7-c554-4596-a4be-ac7c7008ed8a-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.580460 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.580438 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps" pod "68a557d7-c554-4596-a4be-ac7c7008ed8a" (UID: "68a557d7-c554-4596-a4be-ac7c7008ed8a"). InnerVolumeSpecName "kube-api-access-qrrps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:19.614610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.614589 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:19.680218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.680130 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67\" (UniqueName: \"kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67\") pod \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " Oct 31 01:30:19.680366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.680285 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca\") pod \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\" (UID: \"0e3ea459-55d9-4c02-aaba-af8dd03f3406\") " Oct 31 01:30:19.680644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.680622 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca" pod "0e3ea459-55d9-4c02-aaba-af8dd03f3406" (UID: "0e3ea459-55d9-4c02-aaba-af8dd03f3406"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:19.680824 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.680806 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrps\" (UniqueName: \"kubernetes.io/projected/68a557d7-c554-4596-a4be-ac7c7008ed8a-kube-api-access-qrrps\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.680910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.680832 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e3ea459-55d9-4c02-aaba-af8dd03f3406-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.682392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.682373 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67" pod "0e3ea459-55d9-4c02-aaba-af8dd03f3406" (UID: "0e3ea459-55d9-4c02-aaba-af8dd03f3406"). InnerVolumeSpecName "kube-api-access-fpr67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:19.781819 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.781787 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpr67\" (UniqueName: \"kubernetes.io/projected/0e3ea459-55d9-4c02-aaba-af8dd03f3406-kube-api-access-fpr67\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:19.819851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.819813 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd"] Oct 31 01:30:19.823415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:19.823392 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f57e85-openshift-default-748d65db89-5wlsd"] Oct 31 01:30:20.154633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.154594 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c"] Oct 31 01:30:20.158462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.158430 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890170-flf6c"] Oct 31 01:30:20.166464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.166439 5756 generic.go:356] "Generic (PLEG): container finished" podID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerID="50a909f28ec2c412ddfcf0002313f473312340257a5262707b2a6c6f1237f4cb" exitCode=60 Oct 31 01:30:20.166601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.166516 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerDied","Data":"50a909f28ec2c412ddfcf0002313f473312340257a5262707b2a6c6f1237f4cb"} Oct 31 01:30:20.166601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.166576 5756 scope.go:122] "RemoveContainer" containerID="a5e87bf6fabbdc3bde9bfc3e7e565b4168d0fa455d5c35f1ef4c081f7f62cb7f" Oct 31 01:30:20.167051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.167020 5756 scope.go:122] "RemoveContainer" containerID="50a909f28ec2c412ddfcf0002313f473312340257a5262707b2a6c6f1237f4cb" Oct 31 01:30:20.167400 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:20.167374 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86_odh-ai-gateway-infra(bf7793d2-a85d-4c6f-814e-58ff1659b154)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" Oct 31 01:30:20.168610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.168592 5756 generic.go:356] "Generic (PLEG): container finished" podID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerID="8456f8195d0727f6ccfcba9493b2e121c14877cb9a69a316b22a77a6efe51e02" exitCode=60 Oct 31 01:30:20.168695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.168662 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerDied","Data":"8456f8195d0727f6ccfcba9493b2e121c14877cb9a69a316b22a77a6efe51e02"} Oct 31 01:30:20.169060 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.169043 5756 scope.go:122] "RemoveContainer" containerID="8456f8195d0727f6ccfcba9493b2e121c14877cb9a69a316b22a77a6efe51e02" Oct 31 01:30:20.169354 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:20.169333 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr_odh-ai-gateway-infra(9a97b71f-f33b-45ad-8ce8-093b120d13a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" Oct 31 01:30:20.170728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.170710 5756 generic.go:356] "Generic (PLEG): container finished" podID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerID="f607e046f17a7c78f7587bf44972dda05007d5978bb864819954a4a1d0fe6282" exitCode=60 Oct 31 01:30:20.170845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.170760 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerDied","Data":"f607e046f17a7c78f7587bf44972dda05007d5978bb864819954a4a1d0fe6282"} Oct 31 01:30:20.171160 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.171142 5756 scope.go:122] "RemoveContainer" containerID="f607e046f17a7c78f7587bf44972dda05007d5978bb864819954a4a1d0fe6282" Oct 31 01:30:20.171463 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:20.171440 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp_odh-ai-gateway-infra(833600e3-cf61-4dfc-b2f5-00b2dfbe3de8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" Oct 31 01:30:20.172426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.172406 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" Oct 31 01:30:20.172526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.172424 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s" event={"ID":"0e3ea459-55d9-4c02-aaba-af8dd03f3406","Type":"ContainerDied","Data":"7bf8b7ac43a32dabf3f6f7da3acbfbcdc7cf7d6412313357c7c8492de4d7a921"} Oct 31 01:30:20.173918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.173887 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" event={"ID":"68a557d7-c554-4596-a4be-ac7c7008ed8a","Type":"ContainerDied","Data":"19ae2d09cbfc57ff6b5b3ba536cc221fa1fe30394abe2c2e90b88da83d050531"} Oct 31 01:30:20.173918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.173912 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n" Oct 31 01:30:20.182573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.182557 5756 scope.go:122] "RemoveContainer" containerID="75ac9100e1222b341194519e0bc1f6d11bde950aa439712059570f2b4f527ea1" Oct 31 01:30:20.195872 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.195840 5756 scope.go:122] "RemoveContainer" containerID="8cc849ea5d7d6c281bdea2a18ac7a84f016292264c3afa8b60a414ab1c732498" Oct 31 01:30:20.208849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.208831 5756 scope.go:122] "RemoveContainer" containerID="5df3f42428ea373ea5f967662ff3eb6efcc0f8667f477da75f2f07ee2cbfd665" Oct 31 01:30:20.217601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.217579 5756 scope.go:122] "RemoveContainer" containerID="7e371c89f46a3311bd46e4433012d42f26332a9735bfb7fa9625d44943067d67" Oct 31 01:30:20.314301 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.314270 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" path="/var/lib/kubelet/pods/10abdb0d-7e36-4d79-bd39-4f82b09560df/volumes" Oct 31 01:30:20.314744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.314731 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" path="/var/lib/kubelet/pods/ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec/volumes" Oct 31 01:30:20.315128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.315113 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" path="/var/lib/kubelet/pods/e8d4176d-9f4a-4841-911b-adeeee24c7db/volumes" Oct 31 01:30:20.321431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.321403 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp"] Oct 31 01:30:20.434286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.434206 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr"] Oct 31 01:30:20.491983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.491938 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s"] Oct 31 01:30:20.495792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.495765 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890170-4rs6s"] Oct 31 01:30:20.942920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.942883 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n"] Oct 31 01:30:20.946745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:20.946708 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890170-r2z2n"] Oct 31 01:30:21.054584 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.054555 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86"] Oct 31 01:30:21.366839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.366812 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:21.414193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.414172 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:21.418006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.417983 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:21.514276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514199 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca\") pod \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " Oct 31 01:30:21.514276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514237 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca\") pod \"bf7793d2-a85d-4c6f-814e-58ff1659b154\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " Oct 31 01:30:21.514276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514276 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca\") pod \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " Oct 31 01:30:21.514512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514306 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5\" (UniqueName: \"kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5\") pod \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\" (UID: \"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8\") " Oct 31 01:30:21.514512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514341 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7\") pod \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\" (UID: \"9a97b71f-f33b-45ad-8ce8-093b120d13a8\") " Oct 31 01:30:21.514512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514368 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m\" (UniqueName: \"kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m\") pod \"bf7793d2-a85d-4c6f-814e-58ff1659b154\" (UID: \"bf7793d2-a85d-4c6f-814e-58ff1659b154\") " Oct 31 01:30:21.514684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514647 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca" pod "bf7793d2-a85d-4c6f-814e-58ff1659b154" (UID: "bf7793d2-a85d-4c6f-814e-58ff1659b154"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:21.514746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514647 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca" pod "9a97b71f-f33b-45ad-8ce8-093b120d13a8" (UID: "9a97b71f-f33b-45ad-8ce8-093b120d13a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:21.514746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514701 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca" pod "833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" (UID: "833600e3-cf61-4dfc-b2f5-00b2dfbe3de8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:21.514895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514880 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a97b71f-f33b-45ad-8ce8-093b120d13a8-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.514951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514899 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7793d2-a85d-4c6f-814e-58ff1659b154-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.514951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.514914 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.516629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.516600 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m" pod "bf7793d2-a85d-4c6f-814e-58ff1659b154" (UID: "bf7793d2-a85d-4c6f-814e-58ff1659b154"). InnerVolumeSpecName "kube-api-access-gx66m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:21.517194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.517171 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7" pod "9a97b71f-f33b-45ad-8ce8-093b120d13a8" (UID: "9a97b71f-f33b-45ad-8ce8-093b120d13a8"). InnerVolumeSpecName "kube-api-access-zx8h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:21.517268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.517216 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5" pod "833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" (UID: "833600e3-cf61-4dfc-b2f5-00b2dfbe3de8"). InnerVolumeSpecName "kube-api-access-mhnh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:21.616259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.616228 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhnh5\" (UniqueName: \"kubernetes.io/projected/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8-kube-api-access-mhnh5\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.616259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.616259 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx8h7\" (UniqueName: \"kubernetes.io/projected/9a97b71f-f33b-45ad-8ce8-093b120d13a8-kube-api-access-zx8h7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:21.616457 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:21.616270 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx66m\" (UniqueName: \"kubernetes.io/projected/bf7793d2-a85d-4c6f-814e-58ff1659b154-kube-api-access-gx66m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:22.198523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.198490 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" Oct 31 01:30:22.198727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.198521 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr" event={"ID":"9a97b71f-f33b-45ad-8ce8-093b120d13a8","Type":"ContainerDied","Data":"c5c976716cd37643ca8004234f3ddbef7466054538b59a8d2786409250d1234e"} Oct 31 01:30:22.198727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.198598 5756 scope.go:122] "RemoveContainer" containerID="8456f8195d0727f6ccfcba9493b2e121c14877cb9a69a316b22a77a6efe51e02" Oct 31 01:30:22.199904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.199841 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" Oct 31 01:30:22.199904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.199887 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp" event={"ID":"833600e3-cf61-4dfc-b2f5-00b2dfbe3de8","Type":"ContainerDied","Data":"f446c5587b4d6bad4ae3612713427779b3f92262c2d404cad050ca8eac6b432a"} Oct 31 01:30:22.204627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.204604 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" event={"ID":"bf7793d2-a85d-4c6f-814e-58ff1659b154","Type":"ContainerDied","Data":"08007962d855dd3fc7476bb22e65addfaf7d863ffb35bdccede44fe8b499d858"} Oct 31 01:30:22.204776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.204612 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86" Oct 31 01:30:22.210794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.210770 5756 scope.go:122] "RemoveContainer" containerID="f607e046f17a7c78f7587bf44972dda05007d5978bb864819954a4a1d0fe6282" Oct 31 01:30:22.219833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.219817 5756 scope.go:122] "RemoveContainer" containerID="50a909f28ec2c412ddfcf0002313f473312340257a5262707b2a6c6f1237f4cb" Oct 31 01:30:22.311617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.311579 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" path="/var/lib/kubelet/pods/0e3ea459-55d9-4c02-aaba-af8dd03f3406/volumes" Oct 31 01:30:22.312155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.312135 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" path="/var/lib/kubelet/pods/68a557d7-c554-4596-a4be-ac7c7008ed8a/volumes" Oct 31 01:30:22.363820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.363786 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86"] Oct 31 01:30:22.367665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.367641 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-d6b8ff-29890170-29n86"] Oct 31 01:30:22.480898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.480806 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp"] Oct 31 01:30:22.485048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.485022 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-02c8fd-29890170-5q7zp"] Oct 31 01:30:22.597215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.597158 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr"] Oct 31 01:30:22.599223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:22.599175 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890170-kmdzr"] Oct 31 01:30:24.311220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:24.311190 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" path="/var/lib/kubelet/pods/833600e3-cf61-4dfc-b2f5-00b2dfbe3de8/volumes" Oct 31 01:30:24.311624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:24.311576 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" path="/var/lib/kubelet/pods/9a97b71f-f33b-45ad-8ce8-093b120d13a8/volumes" Oct 31 01:30:24.311958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:24.311944 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" path="/var/lib/kubelet/pods/bf7793d2-a85d-4c6f-814e-58ff1659b154/volumes" Oct 31 01:30:24.852141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:24.852112 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98"] Oct 31 01:30:24.852356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:24.852314 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" containerID="cri-o://c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57" gracePeriod=30 Oct 31 01:30:25.165523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:25.165442 5756 patch_prober.go:29] interesting pod/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:30:25.165523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:25.165513 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:30:30.137685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.137655 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:30:30.236605 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236574 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.236605 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236607 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.236935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236644 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.236935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236671 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.236935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236726 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.236935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236875 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:30.236935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236892 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236963 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.236990 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237064 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237092 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237135 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s\") pod \"34b281e1-9c63-443a-923f-86f06b782f29\" (UID: \"34b281e1-9c63-443a-923f-86f06b782f29\") " Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237210 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:30.237225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237226 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:30.237769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237595 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.237769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237610 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.237769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237619 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.237769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237628 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34b281e1-9c63-443a-923f-86f06b782f29-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.237769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.237637 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.239305 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.239277 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:30.239414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.239302 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "kube-api-access-fm45s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:30.239719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.239691 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:30.239719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.239698 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo" pod "34b281e1-9c63-443a-923f-86f06b782f29" (UID: "34b281e1-9c63-443a-923f-86f06b782f29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:30:30.267185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.267153 5756 generic.go:356] "Generic (PLEG): container finished" podID="34b281e1-9c63-443a-923f-86f06b782f29" containerID="c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57" exitCode=0 Oct 31 01:30:30.267328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.267212 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" Oct 31 01:30:30.267328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.267230 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" event={"ID":"34b281e1-9c63-443a-923f-86f06b782f29","Type":"ContainerDied","Data":"c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57"} Oct 31 01:30:30.267328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.267307 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98" event={"ID":"34b281e1-9c63-443a-923f-86f06b782f29","Type":"ContainerDied","Data":"f582f4a1fcc4a1a94297c2d7119eb1671180f6a5be06e5b875a74c49f60b4c4f"} Oct 31 01:30:30.267431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.267340 5756 scope.go:122] "RemoveContainer" containerID="c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57" Oct 31 01:30:30.278668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.278650 5756 scope.go:122] "RemoveContainer" containerID="930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4" Oct 31 01:30:30.288461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.288441 5756 scope.go:122] "RemoveContainer" containerID="c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57" Oct 31 01:30:30.288760 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:30.288737 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57\": container with ID starting with c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57 not found: ID does not exist" containerID="c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57" Oct 31 01:30:30.288817 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.288769 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57"} err="failed to get container status \"c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57\": rpc error: code = NotFound desc = could not find container \"c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57\": container with ID starting with c09cc3e54fef76a8582681e4c6619ba01044053c3648185ce49b4224deb7af57 not found: ID does not exist" Oct 31 01:30:30.288817 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.288786 5756 scope.go:122] "RemoveContainer" containerID="930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4" Oct 31 01:30:30.289101 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:30.289082 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4\": container with ID starting with 930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4 not found: ID does not exist" containerID="930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4" Oct 31 01:30:30.289155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.289108 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4"} err="failed to get container status \"930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4\": rpc error: code = NotFound desc = could not find container \"930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4\": container with ID starting with 930e86c49ff2fbd79e35a2fa83e32bd57d87ea7be78017e5f3991a44ab54aca4 not found: ID does not exist" Oct 31 01:30:30.338632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.338598 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34b281e1-9c63-443a-923f-86f06b782f29-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.338632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.338632 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm45s\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-kube-api-access-fm45s\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.338842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.338651 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34b281e1-9c63-443a-923f-86f06b782f29-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.338842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.338663 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34b281e1-9c63-443a-923f-86f06b782f29-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:30.413992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.413834 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98"] Oct 31 01:30:30.416607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:30.416583 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-88d9b8-openshift-default-7d595b8b79-ttb98"] Oct 31 01:30:32.067377 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:32.067342 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:30:32.312040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:32.312004 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b281e1-9c63-443a-923f-86f06b782f29" path="/var/lib/kubelet/pods/34b281e1-9c63-443a-923f-86f06b782f29/volumes" Oct 31 01:30:35.701616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.701577 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2"] Oct 31 01:30:35.702106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702092 5756 cpu_manager.go:414] "containerMap: removing container" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702107 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702117 5756 cpu_manager.go:414] "containerMap: removing container" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702123 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702131 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702136 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702145 5756 cpu_manager.go:414] "containerMap: removing container" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702150 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" Oct 31 01:30:35.702156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702156 5756 cpu_manager.go:414] "containerMap: removing container" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702161 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702168 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702172 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702184 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702189 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702195 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702199 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702209 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702213 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702221 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702225 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702233 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702236 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702246 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702249 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702255 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702259 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702267 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702271 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702277 5756 cpu_manager.go:414] "containerMap: removing container" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702281 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702289 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702293 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702298 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702302 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702308 5756 cpu_manager.go:414] "containerMap: removing container" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702312 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702322 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702326 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702332 5756 cpu_manager.go:414] "containerMap: removing container" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702335 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702341 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702346 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702364 5756 cpu_manager.go:414] "containerMap: removing container" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702368 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702377 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702381 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702387 5756 cpu_manager.go:414] "containerMap: removing container" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702392 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.702382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702401 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e0382435-c90b-452c-a93f-30d2741c65d8" containerName="main" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702406 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0382435-c90b-452c-a93f-30d2741c65d8" containerName="main" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702490 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702499 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702505 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702510 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702515 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702521 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702528 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702534 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a97b71f-f33b-45ad-8ce8-093b120d13a8" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702539 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702544 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702549 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702554 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d4176d-9f4a-4841-911b-adeeee24c7db" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702560 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702568 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702573 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702577 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702583 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a557d7-c554-4596-a4be-ac7c7008ed8a" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702588 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0382435-c90b-452c-a93f-30d2741c65d8" containerName="main" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702593 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702598 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702603 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702607 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="10abdb0d-7e36-4d79-bd39-4f82b09560df" containerName="istio-proxy" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702612 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="833600e3-cf61-4dfc-b2f5-00b2dfbe3de8" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702618 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702623 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerName="cleanup" Oct 31 01:30:35.703564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.702629 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9d753f-74ee-4631-bb1d-6e51d584aed9" containerName="cleanup" Oct 31 01:30:35.707516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.707498 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.762818 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.762790 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2"] Oct 31 01:30:35.808737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.808704 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:35.808935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.808753 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:30:35.810344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.810320 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.810487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.810472 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbxxx\" (UniqueName: \"kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.810533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.810506 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.911946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.911915 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbxxx\" (UniqueName: \"kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.912136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.911957 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.912136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.912053 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.912632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.912607 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.914546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.914526 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:35.968128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:35.968060 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbxxx\" (UniqueName: \"kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx\") pod \"maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:36.021404 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.021374 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:36.396812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.396785 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2"] Oct 31 01:30:36.398230 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:36.398197 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb764c4f_dac2_492e_932c_662a88d8d558.slice/crio-6883c3d2cffd61d1d30d59bd90a54676c480c163ecde0309538ca64fe01afebb WatchSource:0}: Error finding container 6883c3d2cffd61d1d30d59bd90a54676c480c163ecde0309538ca64fe01afebb: Status 404 returned error can't find the container with id 6883c3d2cffd61d1d30d59bd90a54676c480c163ecde0309538ca64fe01afebb Oct 31 01:30:36.730087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730055 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj"] Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730570 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730581 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7793d2-a85d-4c6f-814e-58ff1659b154" containerName="cleanup" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730598 5756 cpu_manager.go:414] "containerMap: removing container" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730603 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b281e1-9c63-443a-923f-86f06b782f29" containerName="istio-proxy" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730623 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730628 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730715 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccac6f4e-4ee2-4c71-aeaf-cf8fdd7204ec" containerName="cleanup" Oct 31 01:30:36.730721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.730724 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3ea459-55d9-4c02-aaba-af8dd03f3406" containerName="cleanup" Oct 31 01:30:36.735470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.735445 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.790317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.790287 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj"] Oct 31 01:30:36.822508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.822480 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.822677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.822611 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.822677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.822666 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm2qw\" (UniqueName: \"kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.924022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.923943 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.924187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.924036 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.924187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.924122 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm2qw\" (UniqueName: \"kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.925131 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.925105 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.926582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.926560 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:36.980469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:36.980444 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm2qw\" (UniqueName: \"kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw\") pod \"maas-api-revoke-sub-2c8a90d2-w4qcj\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:37.055910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:37.051185 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:37.319920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:37.319884 5756 generic.go:356] "Generic (PLEG): container finished" podID="cb764c4f-dac2-492e-932c-662a88d8d558" containerID="af5278d7b78442ece74fbfc381b857095d838bb49a9536dc924e86d6d4331cb4" exitCode=0 Oct 31 01:30:37.320076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:37.319924 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" event={"ID":"cb764c4f-dac2-492e-932c-662a88d8d558","Type":"ContainerDied","Data":"af5278d7b78442ece74fbfc381b857095d838bb49a9536dc924e86d6d4331cb4"} Oct 31 01:30:37.320076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:37.319981 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" event={"ID":"cb764c4f-dac2-492e-932c-662a88d8d558","Type":"ContainerStarted","Data":"6883c3d2cffd61d1d30d59bd90a54676c480c163ecde0309538ca64fe01afebb"} Oct 31 01:30:37.345599 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:30:37.345574 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb764c4f_dac2_492e_932c_662a88d8d558.slice/crio-af5278d7b78442ece74fbfc381b857095d838bb49a9536dc924e86d6d4331cb4.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:30:37.429490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:37.429465 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj"] Oct 31 01:30:37.430975 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:37.430950 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35c015a_efa9_436f_be81_6216594194ed.slice/crio-eb13a18a12d367e41841a1ed31ee30ead930c50542898bc8bcbe341dd5ab834c WatchSource:0}: Error finding container eb13a18a12d367e41841a1ed31ee30ead930c50542898bc8bcbe341dd5ab834c: Status 404 returned error can't find the container with id eb13a18a12d367e41841a1ed31ee30ead930c50542898bc8bcbe341dd5ab834c Oct 31 01:30:38.328546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.328493 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" event={"ID":"f35c015a-efa9-436f-be81-6216594194ed","Type":"ContainerStarted","Data":"af31e4dd8f375370fa38c60d3c92fd66967dc68de05961ff2600d2a37cde6114"} Oct 31 01:30:38.329023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.328561 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" event={"ID":"f35c015a-efa9-436f-be81-6216594194ed","Type":"ContainerStarted","Data":"eb13a18a12d367e41841a1ed31ee30ead930c50542898bc8bcbe341dd5ab834c"} Oct 31 01:30:38.478616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.478593 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:38.483749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.483703 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" podStartSLOduration=2.483690064 podStartE2EDuration="2.483690064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:38.481900825 +0000 UTC m=+2010.288868370" watchObservedRunningTime="2026-10-31 01:30:38.483690064 +0000 UTC m=+2010.290657607" Oct 31 01:30:38.544268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.544235 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca\") pod \"cb764c4f-dac2-492e-932c-662a88d8d558\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " Oct 31 01:30:38.544268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.544274 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx\" (UniqueName: \"kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx\") pod \"cb764c4f-dac2-492e-932c-662a88d8d558\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " Oct 31 01:30:38.544471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.544440 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca\") pod \"cb764c4f-dac2-492e-932c-662a88d8d558\" (UID: \"cb764c4f-dac2-492e-932c-662a88d8d558\") " Oct 31 01:30:38.544656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.544627 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca" pod "cb764c4f-dac2-492e-932c-662a88d8d558" (UID: "cb764c4f-dac2-492e-932c-662a88d8d558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:38.545036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.545008 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:38.546698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.546670 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca" pod "cb764c4f-dac2-492e-932c-662a88d8d558" (UID: "cb764c4f-dac2-492e-932c-662a88d8d558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:38.546787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.546712 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx" pod "cb764c4f-dac2-492e-932c-662a88d8d558" (UID: "cb764c4f-dac2-492e-932c-662a88d8d558"). InnerVolumeSpecName "kube-api-access-kbxxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:38.620269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.620174 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:30:38.646377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.646347 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb764c4f-dac2-492e-932c-662a88d8d558-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:38.646377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:38.646373 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbxxx\" (UniqueName: \"kubernetes.io/projected/cb764c4f-dac2-492e-932c-662a88d8d558-kube-api-access-kbxxx\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:39.336651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:39.336612 5756 generic.go:356] "Generic (PLEG): container finished" podID="f35c015a-efa9-436f-be81-6216594194ed" containerID="af31e4dd8f375370fa38c60d3c92fd66967dc68de05961ff2600d2a37cde6114" exitCode=0 Oct 31 01:30:39.337142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:39.336706 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" event={"ID":"f35c015a-efa9-436f-be81-6216594194ed","Type":"ContainerDied","Data":"af31e4dd8f375370fa38c60d3c92fd66967dc68de05961ff2600d2a37cde6114"} Oct 31 01:30:39.338231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:39.338129 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" event={"ID":"cb764c4f-dac2-492e-932c-662a88d8d558","Type":"ContainerDied","Data":"6883c3d2cffd61d1d30d59bd90a54676c480c163ecde0309538ca64fe01afebb"} Oct 31 01:30:39.338231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:39.338167 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2" Oct 31 01:30:39.338231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:39.338187 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6883c3d2cffd61d1d30d59bd90a54676c480c163ecde0309538ca64fe01afebb" Oct 31 01:30:40.497531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.497504 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:40.571144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.571117 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca\") pod \"f35c015a-efa9-436f-be81-6216594194ed\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " Oct 31 01:30:40.571315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.571190 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca\") pod \"f35c015a-efa9-436f-be81-6216594194ed\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " Oct 31 01:30:40.571315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.571273 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw\" (UniqueName: \"kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw\") pod \"f35c015a-efa9-436f-be81-6216594194ed\" (UID: \"f35c015a-efa9-436f-be81-6216594194ed\") " Oct 31 01:30:40.571568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.571543 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca" pod "f35c015a-efa9-436f-be81-6216594194ed" (UID: "f35c015a-efa9-436f-be81-6216594194ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:40.571784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.571767 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35c015a-efa9-436f-be81-6216594194ed-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:40.573443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.573426 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca" pod "f35c015a-efa9-436f-be81-6216594194ed" (UID: "f35c015a-efa9-436f-be81-6216594194ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:40.573543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.573524 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw" pod "f35c015a-efa9-436f-be81-6216594194ed" (UID: "f35c015a-efa9-436f-be81-6216594194ed"). InnerVolumeSpecName "kube-api-access-hm2qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:40.672761 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.672682 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm2qw\" (UniqueName: \"kubernetes.io/projected/f35c015a-efa9-436f-be81-6216594194ed-kube-api-access-hm2qw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:40.672761 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:40.672711 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35c015a-efa9-436f-be81-6216594194ed-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:41.352279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:41.352244 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" Oct 31 01:30:41.352279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:41.352272 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj" event={"ID":"f35c015a-efa9-436f-be81-6216594194ed","Type":"ContainerDied","Data":"eb13a18a12d367e41841a1ed31ee30ead930c50542898bc8bcbe341dd5ab834c"} Oct 31 01:30:41.352498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:41.352323 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb13a18a12d367e41841a1ed31ee30ead930c50542898bc8bcbe341dd5ab834c" Oct 31 01:30:47.858058 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:47.858017 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr"] Oct 31 01:30:47.906704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:47.906671 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-02c8fd-2vpnr"] Oct 31 01:30:48.312042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:48.311982 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59202344-fb76-4a4d-989c-19b1e071a560" path="/var/lib/kubelet/pods/59202344-fb76-4a4d-989c-19b1e071a560/volumes" Oct 31 01:30:51.045321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:51.045285 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588"] Oct 31 01:30:51.094570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:51.094539 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37914ab1-zh588"] Oct 31 01:30:52.311336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:52.311303 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f242e15-32d2-4f0e-b93f-f77993db4169" path="/var/lib/kubelet/pods/6f242e15-32d2-4f0e-b93f-f77993db4169/volumes" Oct 31 01:30:54.356230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:54.356193 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r"] Oct 31 01:30:54.356789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:54.356449 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" containerID="cri-o://8fec5a7060c7d42fda89c7faf169fd26fb0afc676df124d21780f99f01ec05f9" gracePeriod=30 Oct 31 01:30:55.170030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.169998 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg"] Oct 31 01:30:55.170547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.170532 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cb764c4f-dac2-492e-932c-662a88d8d558" containerName="revoke-keys" Oct 31 01:30:55.170606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.170552 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb764c4f-dac2-492e-932c-662a88d8d558" containerName="revoke-keys" Oct 31 01:30:55.170606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.170597 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f35c015a-efa9-436f-be81-6216594194ed" containerName="revoke-keys" Oct 31 01:30:55.170606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.170603 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35c015a-efa9-436f-be81-6216594194ed" containerName="revoke-keys" Oct 31 01:30:55.170727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.170717 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35c015a-efa9-436f-be81-6216594194ed" containerName="revoke-keys" Oct 31 01:30:55.170762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.170735 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb764c4f-dac2-492e-932c-662a88d8d558" containerName="revoke-keys" Oct 31 01:30:55.174943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.174927 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.194100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.194072 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.194221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.194129 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.194317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.194295 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xh7\" (UniqueName: \"kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.230833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.230801 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg"] Oct 31 01:30:55.276662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.276633 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:30:55.276662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.276634 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:30:55.295910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.295842 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.296066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.295935 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.296066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.296021 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xh7\" (UniqueName: \"kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.296447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.296427 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.298536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.298512 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.352081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.352051 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xh7\" (UniqueName: \"kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7\") pod \"maas-api-revoke-keys-e2e-api-b-7870b4-599qg\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.489317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.489228 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:55.870757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:55.870733 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg"] Oct 31 01:30:55.872611 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:30:55.872582 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5232c77_f614_422f_90d0_4780f8340346.slice/crio-505e91644ec4e8ff37df3b9367484eb7588143398bbc2294c42b103ac15b3dce WatchSource:0}: Error finding container 505e91644ec4e8ff37df3b9367484eb7588143398bbc2294c42b103ac15b3dce: Status 404 returned error can't find the container with id 505e91644ec4e8ff37df3b9367484eb7588143398bbc2294c42b103ac15b3dce Oct 31 01:30:56.487214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:56.487122 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" event={"ID":"b5232c77-f614-422f-90d0-4780f8340346","Type":"ContainerStarted","Data":"916cc2ce6e3ecf5a79ad018a7321466143d7cbb1c759cd1e829cdec8f57ae417"} Oct 31 01:30:56.487214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:56.487180 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" event={"ID":"b5232c77-f614-422f-90d0-4780f8340346","Type":"ContainerStarted","Data":"505e91644ec4e8ff37df3b9367484eb7588143398bbc2294c42b103ac15b3dce"} Oct 31 01:30:56.599295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:56.599182 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" podStartSLOduration=1.599166179 podStartE2EDuration="1.599166179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:30:56.598375736 +0000 UTC m=+2028.405343280" watchObservedRunningTime="2026-10-31 01:30:56.599166179 +0000 UTC m=+2028.406133722" Oct 31 01:30:57.495148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:57.495108 5756 generic.go:356] "Generic (PLEG): container finished" podID="b5232c77-f614-422f-90d0-4780f8340346" containerID="916cc2ce6e3ecf5a79ad018a7321466143d7cbb1c759cd1e829cdec8f57ae417" exitCode=0 Oct 31 01:30:57.495319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:57.495190 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" event={"ID":"b5232c77-f614-422f-90d0-4780f8340346","Type":"ContainerDied","Data":"916cc2ce6e3ecf5a79ad018a7321466143d7cbb1c759cd1e829cdec8f57ae417"} Oct 31 01:30:58.645591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.645567 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:58.738498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.738470 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca\") pod \"b5232c77-f614-422f-90d0-4780f8340346\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " Oct 31 01:30:58.738666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.738639 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca\") pod \"b5232c77-f614-422f-90d0-4780f8340346\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " Oct 31 01:30:58.738723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.738664 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7\" (UniqueName: \"kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7\") pod \"b5232c77-f614-422f-90d0-4780f8340346\" (UID: \"b5232c77-f614-422f-90d0-4780f8340346\") " Oct 31 01:30:58.738823 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.738801 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca" pod "b5232c77-f614-422f-90d0-4780f8340346" (UID: "b5232c77-f614-422f-90d0-4780f8340346"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:58.739185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.739164 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5232c77-f614-422f-90d0-4780f8340346-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:58.740932 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.740912 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca" pod "b5232c77-f614-422f-90d0-4780f8340346" (UID: "b5232c77-f614-422f-90d0-4780f8340346"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:30:58.741021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.740977 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7" pod "b5232c77-f614-422f-90d0-4780f8340346" (UID: "b5232c77-f614-422f-90d0-4780f8340346"). InnerVolumeSpecName "kube-api-access-n5xh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:58.840442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.840406 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5232c77-f614-422f-90d0-4780f8340346-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:58.840442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:58.840434 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5xh7\" (UniqueName: \"kubernetes.io/projected/b5232c77-f614-422f-90d0-4780f8340346-kube-api-access-n5xh7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.510995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.510964 5756 generic.go:356] "Generic (PLEG): container finished" podID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerID="8fec5a7060c7d42fda89c7faf169fd26fb0afc676df124d21780f99f01ec05f9" exitCode=0 Oct 31 01:30:59.511162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.511037 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" event={"ID":"c2c06d50-b4bf-4682-b178-ed2886ed5f85","Type":"ContainerDied","Data":"8fec5a7060c7d42fda89c7faf169fd26fb0afc676df124d21780f99f01ec05f9"} Oct 31 01:30:59.511162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.511097 5756 scope.go:122] "RemoveContainer" containerID="452b80215b19bc3ccce3a4599ef6253a56073be7b20708bc2a3f3f17b4398a15" Oct 31 01:30:59.517219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.513326 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" event={"ID":"b5232c77-f614-422f-90d0-4780f8340346","Type":"ContainerDied","Data":"505e91644ec4e8ff37df3b9367484eb7588143398bbc2294c42b103ac15b3dce"} Oct 31 01:30:59.517219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.513430 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505e91644ec4e8ff37df3b9367484eb7588143398bbc2294c42b103ac15b3dce" Oct 31 01:30:59.517219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.513522 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg" Oct 31 01:30:59.615572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.615548 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:30:59.650986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.650917 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.650986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.650957 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651061 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651109 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651142 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651208 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651201 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651237 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651325 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651405 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo\") pod \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\" (UID: \"c2c06d50-b4bf-4682-b178-ed2886ed5f85\") " Oct 31 01:30:59.651444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651428 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651642 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651649 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651658 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.651985 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.652010 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.652019 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.652030 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.652127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.652043 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.653736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.653709 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:30:59.653846 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.653717 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:30:59.654107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.654082 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:59.654201 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.654108 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf" pod "c2c06d50-b4bf-4682-b178-ed2886ed5f85" (UID: "c2c06d50-b4bf-4682-b178-ed2886ed5f85"). InnerVolumeSpecName "kube-api-access-6cpkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:30:59.752998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.752964 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.752998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.752990 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.752998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.753001 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2c06d50-b4bf-4682-b178-ed2886ed5f85-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:30:59.753300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:30:59.753010 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cpkf\" (UniqueName: \"kubernetes.io/projected/c2c06d50-b4bf-4682-b178-ed2886ed5f85-kube-api-access-6cpkf\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:00.520953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:00.520919 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" Oct 31 01:31:00.521130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:00.520959 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r" event={"ID":"c2c06d50-b4bf-4682-b178-ed2886ed5f85","Type":"ContainerDied","Data":"7aa9e62126492d642045aecc34d2d293753fbe5de5ef3fa5c718a0db6618f33e"} Oct 31 01:31:00.521130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:00.521016 5756 scope.go:122] "RemoveContainer" containerID="8fec5a7060c7d42fda89c7faf169fd26fb0afc676df124d21780f99f01ec05f9" Oct 31 01:31:00.734581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:00.734491 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r"] Oct 31 01:31:00.784169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:00.784137 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-02c8fd-openshift-default-5b4875c4d7-4r55r"] Oct 31 01:31:02.066905 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:31:02.066850 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:31:02.311620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:02.311591 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" path="/var/lib/kubelet/pods/c2c06d50-b4bf-4682-b178-ed2886ed5f85/volumes" Oct 31 01:31:09.574925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:09.574894 5756 scope.go:122] "RemoveContainer" containerID="7f95f303608648880bffc55e68d30c2c0d6f807617b51d9e8f15754d5cf623fd" Oct 31 01:31:09.585567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:09.585546 5756 scope.go:122] "RemoveContainer" containerID="f0943cc27a81221b8ae09cc60a148a4eefac75bbf6c082aeb9758a01ee741e4b" Oct 31 01:31:15.823111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:15.823080 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6"] Oct 31 01:31:15.823496 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:15.823291 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" containerID="cri-o://f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec" gracePeriod=30 Oct 31 01:31:18.069255 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:18.069218 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56"] Oct 31 01:31:18.069763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:18.069447 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" containerID="cri-o://36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102" gracePeriod=30 Oct 31 01:31:21.096560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.096503 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:31:21.223175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223144 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223183 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223231 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223293 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223426 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223538 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223575 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223596 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:21.223758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223665 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223696 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:21.223895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223793 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data\") pod \"c1200043-f2f0-45c1-8620-ac65307b0bdc\" (UID: \"c1200043-f2f0-45c1-8620-ac65307b0bdc\") " Oct 31 01:31:21.223895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223773 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:21.224026 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.223965 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:21.224245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.224066 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:21.224656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.224632 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.224732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.224661 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.224732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.224677 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1200043-f2f0-45c1-8620-ac65307b0bdc-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.224732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.224691 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.224732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.224707 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.225660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.225636 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:21.225750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.225736 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:21.225803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.225760 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:31:21.225939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.225921 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n" pod "c1200043-f2f0-45c1-8620-ac65307b0bdc" (UID: "c1200043-f2f0-45c1-8620-ac65307b0bdc"). InnerVolumeSpecName "kube-api-access-56l4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:21.325712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.325679 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56l4n\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-kube-api-access-56l4n\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.325712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.325706 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.325712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.325716 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.326023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.325724 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1200043-f2f0-45c1-8620-ac65307b0bdc-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:21.658271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.658238 5756 generic.go:356] "Generic (PLEG): container finished" podID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerID="f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec" exitCode=0 Oct 31 01:31:21.658440 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.658292 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" Oct 31 01:31:21.658440 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.658324 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" event={"ID":"c1200043-f2f0-45c1-8620-ac65307b0bdc","Type":"ContainerDied","Data":"f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec"} Oct 31 01:31:21.658440 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.658382 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6" event={"ID":"c1200043-f2f0-45c1-8620-ac65307b0bdc","Type":"ContainerDied","Data":"145c19599f9341135deff3005697ff7c0c97037423c140dc34a991dc1b62dd57"} Oct 31 01:31:21.658440 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.658413 5756 scope.go:122] "RemoveContainer" containerID="f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec" Oct 31 01:31:21.668955 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.668930 5756 scope.go:122] "RemoveContainer" containerID="f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86" Oct 31 01:31:21.682072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.682047 5756 scope.go:122] "RemoveContainer" containerID="f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec" Oct 31 01:31:21.682387 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:31:21.682364 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec\": container with ID starting with f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec not found: ID does not exist" containerID="f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec" Oct 31 01:31:21.682464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.682411 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec"} err="failed to get container status \"f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec\": rpc error: code = NotFound desc = could not find container \"f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec\": container with ID starting with f012c20dd0b506e6c6a8eebe7563734f65291b5103e60e462573535827130fec not found: ID does not exist" Oct 31 01:31:21.682464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.682435 5756 scope.go:122] "RemoveContainer" containerID="f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86" Oct 31 01:31:21.682692 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:31:21.682673 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86\": container with ID starting with f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86 not found: ID does not exist" containerID="f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86" Oct 31 01:31:21.682747 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.682698 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86"} err="failed to get container status \"f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86\": rpc error: code = NotFound desc = could not find container \"f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86\": container with ID starting with f4be24f54b93f4d36c8345be0f37f0fd30dd3f5f7fecdf67a4dcfa9e58411c86 not found: ID does not exist" Oct 31 01:31:21.822822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.822789 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6"] Oct 31 01:31:21.824844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:21.824822 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-987f2f-openshift-default-58d767c64c-qh8k6"] Oct 31 01:31:22.315885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:22.315748 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" path="/var/lib/kubelet/pods/c1200043-f2f0-45c1-8620-ac65307b0bdc/volumes" Oct 31 01:31:23.336919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.336827 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:31:23.350173 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350148 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350188 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350247 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350330 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350375 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350415 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:23.350531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350428 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350486 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350552 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350590 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:23.350685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350607 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket\") pod \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\" (UID: \"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f\") " Oct 31 01:31:23.350685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350607 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:23.350965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.350930 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:23.351245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.351146 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:23.351338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.351314 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.351409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.351335 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.351409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.351355 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.351409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.351369 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.351409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.351382 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.352601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.352576 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:23.352753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.352738 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:23.353029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.353009 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:31:23.354062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.354042 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8" pod "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" (UID: "d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f"). InnerVolumeSpecName "kube-api-access-rl8v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:23.452654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.452618 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.452654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.452655 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl8v8\" (UniqueName: \"kubernetes.io/projected/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-kube-api-access-rl8v8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.452901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.452669 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.452901 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.452681 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:23.674586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.674448 5756 generic.go:356] "Generic (PLEG): container finished" podID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerID="36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102" exitCode=0 Oct 31 01:31:23.674810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.674597 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" event={"ID":"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f","Type":"ContainerDied","Data":"36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102"} Oct 31 01:31:23.674810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.674654 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" event={"ID":"d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f","Type":"ContainerDied","Data":"bc7bafd12224cda21a3725ad537746c5db36b1d8c2e0eb27b574ddef66b56d00"} Oct 31 01:31:23.674810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.674686 5756 scope.go:122] "RemoveContainer" containerID="36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102" Oct 31 01:31:23.675046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.674834 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56" Oct 31 01:31:23.693972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.693945 5756 scope.go:122] "RemoveContainer" containerID="36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102" Oct 31 01:31:23.694249 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:31:23.694229 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102\": container with ID starting with 36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102 not found: ID does not exist" containerID="36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102" Oct 31 01:31:23.694295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.694257 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102"} err="failed to get container status \"36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102\": rpc error: code = NotFound desc = could not find container \"36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102\": container with ID starting with 36033d3440b4aa1bb15fe810118723574494dced2c32bbeb16d61d4c60286102 not found: ID does not exist" Oct 31 01:31:23.837048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.837016 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56"] Oct 31 01:31:23.840283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:23.840259 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-xjd56"] Oct 31 01:31:24.310954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:24.310920 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" path="/var/lib/kubelet/pods/d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f/volumes" Oct 31 01:31:31.110543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:31.110509 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:31:32.067024 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:31:32.066984 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:31:35.628603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:35.628550 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2"] Oct 31 01:31:35.677915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:35.677879 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d6b8ff-wpsc2"] Oct 31 01:31:36.311361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:36.311332 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb764c4f-dac2-492e-932c-662a88d8d558" path="/var/lib/kubelet/pods/cb764c4f-dac2-492e-932c-662a88d8d558/volumes" Oct 31 01:31:47.244094 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:47.244012 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp"] Oct 31 01:31:47.244517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:47.244220 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" containerID="cri-o://904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6" gracePeriod=30 Oct 31 01:31:48.547611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:48.547575 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:31:48.548049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:48.547645 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:31:52.490230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.490203 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:31:52.574048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574015 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574068 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574119 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574199 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574242 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574360 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574400 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:52.574563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574410 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:31:52.574563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574497 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:52.574563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574540 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574583 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574608 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:52.574727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574619 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo\") pod \"1451e36d-b999-42b2-a6ea-21bdf88e6798\" (UID: \"1451e36d-b999-42b2-a6ea-21bdf88e6798\") " Oct 31 01:31:52.574920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.574842 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:52.575176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.575152 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.575297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.575178 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.575297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.575192 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1451e36d-b999-42b2-a6ea-21bdf88e6798-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.575297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.575205 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.575297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.575217 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.576664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.576643 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:31:52.576771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.576692 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "kube-api-access-gr9qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:52.577233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.577211 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:31:52.577233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.577228 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token" pod "1451e36d-b999-42b2-a6ea-21bdf88e6798" (UID: "1451e36d-b999-42b2-a6ea-21bdf88e6798"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:31:52.676572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.676538 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.676768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.676608 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9qr\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-kube-api-access-gr9qr\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.676768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.676653 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.676768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.676665 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1451e36d-b999-42b2-a6ea-21bdf88e6798-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:31:52.859827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.859740 5756 generic.go:356] "Generic (PLEG): container finished" podID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerID="904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6" exitCode=0 Oct 31 01:31:52.859827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.859789 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" Oct 31 01:31:52.860067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.859823 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" event={"ID":"1451e36d-b999-42b2-a6ea-21bdf88e6798","Type":"ContainerDied","Data":"904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6"} Oct 31 01:31:52.860067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.859911 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp" event={"ID":"1451e36d-b999-42b2-a6ea-21bdf88e6798","Type":"ContainerDied","Data":"51a7bacd868a79aa053d8f79c881158921e4810dd0585025ea0f7dd194deea86"} Oct 31 01:31:52.860067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.859941 5756 scope.go:122] "RemoveContainer" containerID="904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6" Oct 31 01:31:52.873490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.873469 5756 scope.go:122] "RemoveContainer" containerID="904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6" Oct 31 01:31:52.873782 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:31:52.873764 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6\": container with ID starting with 904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6 not found: ID does not exist" containerID="904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6" Oct 31 01:31:52.873857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:52.873794 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6"} err="failed to get container status \"904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6\": rpc error: code = NotFound desc = could not find container \"904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6\": container with ID starting with 904abe9bf02ec4ae2ac6fa3c38dc077d43be08a3f932b633ac9ecf29917a13a6 not found: ID does not exist" Oct 31 01:31:53.022321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:53.022285 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp"] Oct 31 01:31:53.026546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:53.026520 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-kzqfp"] Oct 31 01:31:54.310578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:31:54.310536 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" path="/var/lib/kubelet/pods/1451e36d-b999-42b2-a6ea-21bdf88e6798/volumes" Oct 31 01:32:02.065929 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:32:02.065893 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:32:05.811086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:05.811057 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg"] Oct 31 01:32:05.861808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:05.861775 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7870b4-599qg"] Oct 31 01:32:06.311615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:06.311579 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5232c77-f614-422f-90d0-4780f8340346" path="/var/lib/kubelet/pods/b5232c77-f614-422f-90d0-4780f8340346/volumes" Oct 31 01:32:08.721245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:08.721209 5756 kubelet.go:1689] "Image garbage collection succeeded" Oct 31 01:32:09.658247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:09.658216 5756 scope.go:122] "RemoveContainer" containerID="d2c6822b1930616642d539cf0f58fb48e232a513a06683541582ae7f8f7db849" Oct 31 01:32:09.667584 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:09.667567 5756 scope.go:122] "RemoveContainer" containerID="a0e5038862404d0577438e99884744b496241ba52d8c6222072aba38fbdcb16a" Oct 31 01:32:09.676176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:09.676156 5756 scope.go:122] "RemoveContainer" containerID="d9cc72e8d3bd91ea2f86c7c08ed52a7188a42c3388c88dc8f0fce50646dd6b96" Oct 31 01:32:32.068013 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:32:32.067970 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:32:36.980328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:36.980295 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:32:39.122543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:39.122501 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf"] Oct 31 01:32:39.123015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:39.122776 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" containerID="cri-o://16ab930dc3504ed07bfaa7fec293927c66292018ef325e3ebca026cbc26da43c" gracePeriod=30 Oct 31 01:32:39.172896 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:39.172835 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv"] Oct 31 01:32:39.173342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:39.173309 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" containerID="cri-o://63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf" gracePeriod=30 Oct 31 01:32:41.792949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:41.792905 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-79dvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:32:41.793427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:41.792972 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:32:41.876834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:41.876797 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-zzqhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:32:41.877026 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:41.876848 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:32:44.241806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.241776 5756 generic.go:356] "Generic (PLEG): container finished" podID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerID="16ab930dc3504ed07bfaa7fec293927c66292018ef325e3ebca026cbc26da43c" exitCode=0 Oct 31 01:32:44.242199 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.241885 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" event={"ID":"c52e8f18-4093-4a3e-8d8f-a502b1adee86","Type":"ContainerDied","Data":"16ab930dc3504ed07bfaa7fec293927c66292018ef325e3ebca026cbc26da43c"} Oct 31 01:32:44.450518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.450490 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:32:44.477263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.477239 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:32:44.516488 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516454 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.516674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516495 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.516674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516523 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.516674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516573 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.516674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516594 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.516674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516614 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.516674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516672 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516698 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516731 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516762 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516790 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516817 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516857 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.516954 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.517041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517018 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517046 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517064 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517087 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517157 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517227 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517261 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517262 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket\") pod \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\" (UID: \"cad7eb15-197a-441b-aaa2-df5bc76d84a6\") " Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517347 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket\") pod \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\" (UID: \"c52e8f18-4093-4a3e-8d8f-a502b1adee86\") " Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517445 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.517470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517454 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:32:44.518026 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.517803 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.518087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518034 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.518170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518139 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518164 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518181 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518196 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518209 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518224 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518237 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518250 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518262 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.518615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.518383 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.519578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.519546 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:44.520250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520224 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.520348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520254 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "kube-api-access-d8bv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:44.520348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520304 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:44.520490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520473 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:32:44.520737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520715 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo" pod "cad7eb15-197a-441b-aaa2-df5bc76d84a6" (UID: "cad7eb15-197a-441b-aaa2-df5bc76d84a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:32:44.520737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520732 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "kube-api-access-99dxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:32:44.520829 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.520816 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo" pod "c52e8f18-4093-4a3e-8d8f-a502b1adee86" (UID: "c52e8f18-4093-4a3e-8d8f-a502b1adee86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:32:44.619799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619766 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.619799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619795 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99dxv\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-kube-api-access-99dxv\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.619799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619806 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.620065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619815 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bv4\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-kube-api-access-d8bv4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.620065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619826 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.620065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619836 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.620065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619844 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52e8f18-4093-4a3e-8d8f-a502b1adee86-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.620065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619852 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:44.620065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:44.619887 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad7eb15-197a-441b-aaa2-df5bc76d84a6-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:32:45.248952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.248915 5756 generic.go:356] "Generic (PLEG): container finished" podID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerID="63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf" exitCode=0 Oct 31 01:32:45.249401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.248964 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" Oct 31 01:32:45.249401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.248998 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" event={"ID":"cad7eb15-197a-441b-aaa2-df5bc76d84a6","Type":"ContainerDied","Data":"63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf"} Oct 31 01:32:45.249401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.249056 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv" event={"ID":"cad7eb15-197a-441b-aaa2-df5bc76d84a6","Type":"ContainerDied","Data":"50d3fb359788af4294e6ca6cd4a1b9614577573133d950b4f6d9e26c81177bcd"} Oct 31 01:32:45.249401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.249085 5756 scope.go:122] "RemoveContainer" containerID="63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf" Oct 31 01:32:45.250617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.250586 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" event={"ID":"c52e8f18-4093-4a3e-8d8f-a502b1adee86","Type":"ContainerDied","Data":"228f074d4abc5731027e57edb3e69d4643b8ac7f82b317e8a3a7b736faa137a3"} Oct 31 01:32:45.250617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.250603 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf" Oct 31 01:32:45.261274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.261136 5756 scope.go:122] "RemoveContainer" containerID="63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf" Oct 31 01:32:45.261536 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:32:45.261513 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf\": container with ID starting with 63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf not found: ID does not exist" containerID="63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf" Oct 31 01:32:45.261592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.261549 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf"} err="failed to get container status \"63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf\": rpc error: code = NotFound desc = could not find container \"63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf\": container with ID starting with 63ea6c1671a7f25aa532806bab8762f194e914376eb60d4ef143e48173911bcf not found: ID does not exist" Oct 31 01:32:45.261592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.261574 5756 scope.go:122] "RemoveContainer" containerID="16ab930dc3504ed07bfaa7fec293927c66292018ef325e3ebca026cbc26da43c" Oct 31 01:32:45.412133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.412056 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv"] Oct 31 01:32:45.461629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.461598 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-79dvv"] Oct 31 01:32:45.522916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.522881 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf"] Oct 31 01:32:45.525926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:45.525904 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-zzqhf"] Oct 31 01:32:46.310044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:46.310012 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" path="/var/lib/kubelet/pods/c52e8f18-4093-4a3e-8d8f-a502b1adee86/volumes" Oct 31 01:32:46.310486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:32:46.310472 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" path="/var/lib/kubelet/pods/cad7eb15-197a-441b-aaa2-df5bc76d84a6/volumes" Oct 31 01:33:02.066722 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:33:02.066684 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:33:05.754391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754355 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll"] Oct 31 01:33:05.754854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754841 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" Oct 31 01:33:05.754854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754855 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754892 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754898 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754908 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b5232c77-f614-422f-90d0-4780f8340346" containerName="revoke-keys" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754914 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5232c77-f614-422f-90d0-4780f8340346" containerName="revoke-keys" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754923 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754928 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754935 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754941 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754955 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754960 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754972 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" Oct 31 01:33:05.754977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754976 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754989 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.754994 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755002 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755006 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755072 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755082 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c06d50-b4bf-4682-b178-ed2886ed5f85" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755090 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755095 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5232c77-f614-422f-90d0-4780f8340346" containerName="revoke-keys" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755100 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad7eb15-197a-441b-aaa2-df5bc76d84a6" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755107 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7f0c5a5-4f65-4f6f-a0c4-93c655ee0d9f" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755115 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52e8f18-4093-4a3e-8d8f-a502b1adee86" containerName="istio-proxy" Oct 31 01:33:05.755328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.755121 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1451e36d-b999-42b2-a6ea-21bdf88e6798" containerName="istio-proxy" Oct 31 01:33:05.758303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.758287 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.839891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.839843 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-0ea963-openshift-default-dockercfg-7xbtx\"" Oct 31 01:33:05.874527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874484 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874539 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6tvx\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874576 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874636 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874667 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874733 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874760 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874844 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.874972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.874909 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.890843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.890813 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll"] Oct 31 01:33:05.976356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976320 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.976570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976405 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.976570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976502 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6tvx\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.976570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976561 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.976774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976636 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.976774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976670 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.976774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976751 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976777 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.976907 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.977080 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.977265 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977413 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.977394 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.977526 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.977597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.977533 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.979582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.979560 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:05.979778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:05.979761 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:06.034181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.034146 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:06.034358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.034240 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6tvx\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx\") pod \"e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:06.073688 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.073657 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:06.512284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.512257 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll"] Oct 31 01:33:06.514780 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:33:06.514754 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949b38b8_99b4_4da4_96d8_9ff1bd5d0557.slice/crio-6af8cde1159b8269840da7c35e352f94c7d14bd33259c8880048344f14775b83 WatchSource:0}: Error finding container 6af8cde1159b8269840da7c35e352f94c7d14bd33259c8880048344f14775b83: Status 404 returned error can't find the container with id 6af8cde1159b8269840da7c35e352f94c7d14bd33259c8880048344f14775b83 Oct 31 01:33:06.517391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.517364 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:33:06.517464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.517424 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:33:06.517464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:06.517450 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:33:07.392176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:07.392138 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" event={"ID":"949b38b8-99b4-4da4-96d8-9ff1bd5d0557","Type":"ContainerStarted","Data":"b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a"} Oct 31 01:33:07.392567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:07.392190 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" event={"ID":"949b38b8-99b4-4da4-96d8-9ff1bd5d0557","Type":"ContainerStarted","Data":"6af8cde1159b8269840da7c35e352f94c7d14bd33259c8880048344f14775b83"} Oct 31 01:33:07.510506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:07.510451 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podStartSLOduration=2.5104363320000003 podStartE2EDuration="2.510436332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:07.508652047 +0000 UTC m=+2159.315619594" watchObservedRunningTime="2026-10-31 01:33:07.510436332 +0000 UTC m=+2159.317403875" Oct 31 01:33:08.074384 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:08.074356 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:08.080139 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:08.080109 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:08.397930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:08.397840 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:08.398784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:08.398763 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:33:28.451174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:28.451085 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:33:32.123428 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:33:32.123380 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:33:34.824297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.824252 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr"] Oct 31 01:33:34.824907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.824892 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1200043-f2f0-45c1-8620-ac65307b0bdc" containerName="istio-proxy" Oct 31 01:33:34.828085 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.828062 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:34.930662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.930621 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f5b0cb\"" Oct 31 01:33:34.930830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.930686 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f5b0cb-dockercfg-qzst2\"" Oct 31 01:33:34.932878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.932843 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr"] Oct 31 01:33:34.936997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.936972 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj7vg\" (UniqueName: \"kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg\") pod \"payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:34.937073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:34.937057 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:35.038196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.038165 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj7vg\" (UniqueName: \"kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg\") pod \"payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:35.038396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.038207 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:35.038939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.038917 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:35.094118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.094048 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj7vg\" (UniqueName: \"kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg\") pod \"payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:35.143394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.143365 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:35.516319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.516291 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr"] Oct 31 01:33:35.518066 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:33:35.518038 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6798b3a4_3452_4c64_8fb7_7b26c628193a.slice/crio-3c08cd604ad7a9b91a74cbb1fb46a1f22e617e3696e5b9acb991e31f33b15f60 WatchSource:0}: Error finding container 3c08cd604ad7a9b91a74cbb1fb46a1f22e617e3696e5b9acb991e31f33b15f60: Status 404 returned error can't find the container with id 3c08cd604ad7a9b91a74cbb1fb46a1f22e617e3696e5b9acb991e31f33b15f60 Oct 31 01:33:35.520126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.520112 5756 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:33:35.569696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:35.569665 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" event={"ID":"6798b3a4-3452-4c64-8fb7-7b26c628193a","Type":"ContainerStarted","Data":"3c08cd604ad7a9b91a74cbb1fb46a1f22e617e3696e5b9acb991e31f33b15f60"} Oct 31 01:33:37.583933 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:37.583897 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" event={"ID":"6798b3a4-3452-4c64-8fb7-7b26c628193a","Type":"ContainerStarted","Data":"f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d"} Oct 31 01:33:37.584324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:37.584015 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:37.698747 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:37.698627 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" podStartSLOduration=1.86715934 podStartE2EDuration="3.698607938s" totalImagesPullingTime="1.831448598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:37.696479949 +0000 UTC m=+2189.503447492" watchObservedRunningTime="2026-10-31 01:33:37.698607938 +0000 UTC m=+2189.505575482" Oct 31 01:33:48.592380 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:48.592350 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:33:51.613414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.613380 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r"] Oct 31 01:33:51.617399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.617378 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.720174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.720132 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7cfab8\"" Oct 31 01:33:51.720347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.720185 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7cfab8-dockercfg-bs9rj\"" Oct 31 01:33:51.722346 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.722324 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r"] Oct 31 01:33:51.730904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.730855 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.731068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.731049 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjl4w\" (UniqueName: \"kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w\") pod \"payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.832543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.832510 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjl4w\" (UniqueName: \"kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w\") pod \"payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.832750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.832609 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.833293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.833274 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.890001 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.889935 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjl4w\" (UniqueName: \"kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w\") pod \"payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:51.930971 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:51.930941 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:52.307096 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:33:52.307061 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5d4b73_f6e4_44ec_9d94_1ad95844a892.slice/crio-6737e391bb00f35bc2f3599369ef0d837402c0200f540c61a502b536eff6c0e1 WatchSource:0}: Error finding container 6737e391bb00f35bc2f3599369ef0d837402c0200f540c61a502b536eff6c0e1: Status 404 returned error can't find the container with id 6737e391bb00f35bc2f3599369ef0d837402c0200f540c61a502b536eff6c0e1 Oct 31 01:33:52.311787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.311758 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r"] Oct 31 01:33:52.328638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.328611 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm"] Oct 31 01:33:52.334417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.334395 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.446953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.440761 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzch\" (UniqueName: \"kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch\") pod \"payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.446953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.443784 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.446953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.444426 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm"] Oct 31 01:33:52.545185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.544772 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.545185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.544849 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzch\" (UniqueName: \"kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch\") pod \"payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.545625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.545597 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.601621 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.601548 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzch\" (UniqueName: \"kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch\") pod \"payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.676652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.676610 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:52.682378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.682347 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" event={"ID":"cb5d4b73-f6e4-44ec-9d94-1ad95844a892","Type":"ContainerStarted","Data":"b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd"} Oct 31 01:33:52.682521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.682406 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" event={"ID":"cb5d4b73-f6e4-44ec-9d94-1ad95844a892","Type":"ContainerStarted","Data":"6737e391bb00f35bc2f3599369ef0d837402c0200f540c61a502b536eff6c0e1"} Oct 31 01:33:52.682521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.682510 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:33:52.800938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:52.800882 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" podStartSLOduration=1.800847132 podStartE2EDuration="1.800847132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:52.799015021 +0000 UTC m=+2204.605982566" watchObservedRunningTime="2026-10-31 01:33:52.800847132 +0000 UTC m=+2204.607814675" Oct 31 01:33:53.059232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:53.059199 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm"] Oct 31 01:33:53.062351 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:33:53.062320 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ac98f5_3a81_4fae_8f27_ed2b043fbdf0.slice/crio-ad8e2e0c6e0e4f8fefd1872ea894097814856e5492462110ed7bf5c5bb3367b8 WatchSource:0}: Error finding container ad8e2e0c6e0e4f8fefd1872ea894097814856e5492462110ed7bf5c5bb3367b8: Status 404 returned error can't find the container with id ad8e2e0c6e0e4f8fefd1872ea894097814856e5492462110ed7bf5c5bb3367b8 Oct 31 01:33:53.690124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:53.690087 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" event={"ID":"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0","Type":"ContainerStarted","Data":"0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d"} Oct 31 01:33:53.690532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:53.690154 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" event={"ID":"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0","Type":"ContainerStarted","Data":"ad8e2e0c6e0e4f8fefd1872ea894097814856e5492462110ed7bf5c5bb3367b8"} Oct 31 01:33:53.690532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:53.690342 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:33:53.807006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:33:53.806945 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" podStartSLOduration=1.806927535 podStartE2EDuration="1.806927535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:53.80418327 +0000 UTC m=+2205.611150817" watchObservedRunningTime="2026-10-31 01:33:53.806927535 +0000 UTC m=+2205.613895079" Oct 31 01:34:02.067907 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:34:02.067841 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:34:03.692397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:03.692371 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:34:04.698692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:04.698661 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:34:08.620593 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:08.620559 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:34:09.750913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:09.750881 5756 scope.go:122] "RemoveContainer" containerID="974aae633f78ab7508e30b80f8dbe915c17a12ed2b935bf1a57bc6aa02a570cb" Oct 31 01:34:09.760144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:09.760122 5756 scope.go:122] "RemoveContainer" containerID="8f81f27a725a3a9e529fb4a1976fe4cbe45479dbfb9a7ae0dc2b64275169c4ce" Oct 31 01:34:19.081592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.081554 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd"] Oct 31 01:34:19.085274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.085247 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.174105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.174071 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzf6r\" (UniqueName: \"kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r\") pod \"payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.174278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.174260 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.188160 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.188132 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b95dca\"" Oct 31 01:34:19.188304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.188183 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b95dca-dockercfg-sk8rm\"" Oct 31 01:34:19.190606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.190586 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd"] Oct 31 01:34:19.275730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.275703 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzf6r\" (UniqueName: \"kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r\") pod \"payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.275922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.275840 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.276546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.276520 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.333702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.333630 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzf6r\" (UniqueName: \"kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r\") pod \"payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.397619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.397588 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:19.777555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.777531 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd"] Oct 31 01:34:19.780302 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:34:19.780269 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cd785d_35b0_4008_a5f4_eb64139aee68.slice/crio-067610332ae07bf3c890ed61227c1fb3072a1c7bdf602c593839b55ee87414f9 WatchSource:0}: Error finding container 067610332ae07bf3c890ed61227c1fb3072a1c7bdf602c593839b55ee87414f9: Status 404 returned error can't find the container with id 067610332ae07bf3c890ed61227c1fb3072a1c7bdf602c593839b55ee87414f9 Oct 31 01:34:19.860204 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:19.860167 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" event={"ID":"e5cd785d-35b0-4008-a5f4-eb64139aee68","Type":"ContainerStarted","Data":"067610332ae07bf3c890ed61227c1fb3072a1c7bdf602c593839b55ee87414f9"} Oct 31 01:34:20.867903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:20.867840 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" event={"ID":"e5cd785d-35b0-4008-a5f4-eb64139aee68","Type":"ContainerStarted","Data":"03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7"} Oct 31 01:34:20.868323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:20.867939 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:20.983165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:20.983097 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" podStartSLOduration=1.983078863 podStartE2EDuration="1.983078863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:20.982278145 +0000 UTC m=+2232.789245689" watchObservedRunningTime="2026-10-31 01:34:20.983078863 +0000 UTC m=+2232.790046410" Oct 31 01:34:31.875332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:31.875303 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:34:32.067351 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:34:32.067316 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:34:35.422309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:35.422276 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:34:46.014726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:46.014645 5756 generic.go:356] "Generic (PLEG): container finished" podID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerID="b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a" exitCode=0 Oct 31 01:34:46.014726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:46.014691 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" event={"ID":"949b38b8-99b4-4da4-96d8-9ff1bd5d0557","Type":"ContainerDied","Data":"b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a"} Oct 31 01:34:46.015250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:46.015052 5756 scope.go:122] "RemoveContainer" containerID="b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a" Oct 31 01:34:46.016581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:46.016548 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:34:46.016701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:46.016649 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:34:46.016771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:46.016700 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:34:47.023130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:47.023099 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" event={"ID":"949b38b8-99b4-4da4-96d8-9ff1bd5d0557","Type":"ContainerStarted","Data":"a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684"} Oct 31 01:34:47.073913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:47.073848 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:34:47.074088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:47.074053 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 01:34:47.074135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:47.074095 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 01:34:48.084150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:48.084109 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 01:34:48.084848 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:48.084814 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 01:34:49.074677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:49.074641 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 01:34:49.074887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:49.074698 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 01:34:50.074324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:50.074289 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 01:34:50.074735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:50.074358 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 01:34:51.074250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:51.074220 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 31 01:34:51.074432 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:51.074281 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 31 01:34:52.141420 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:52.141381 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:34:52.141942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:52.141793 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:34:52.142949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:52.142930 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:34:56.222038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.222009 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb"] Oct 31 01:34:56.226237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.226219 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.283659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.283632 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a31828\"" Oct 31 01:34:56.283833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.283680 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a31828-dockercfg-vng7w\"" Oct 31 01:34:56.320695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.320650 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dq59\" (UniqueName: \"kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59\") pod \"payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.321211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.321006 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.335383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.335353 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb"] Oct 31 01:34:56.422214 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.422175 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.422419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.422360 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dq59\" (UniqueName: \"kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59\") pod \"payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.423092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.423070 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.480396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.480323 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dq59\" (UniqueName: \"kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59\") pod \"payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.541595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.541562 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:56.927808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:56.927785 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb"] Oct 31 01:34:56.929629 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:34:56.929599 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88f9b32_4ce1_411c_a57e_f891e549b821.slice/crio-999abdbc6b866abccffc2d8c2034548ae0550728194eb4ac804fcebd0dcc8a1f WatchSource:0}: Error finding container 999abdbc6b866abccffc2d8c2034548ae0550728194eb4ac804fcebd0dcc8a1f: Status 404 returned error can't find the container with id 999abdbc6b866abccffc2d8c2034548ae0550728194eb4ac804fcebd0dcc8a1f Oct 31 01:34:57.107005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:57.106972 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" event={"ID":"e88f9b32-4ce1-411c-a57e-f891e549b821","Type":"ContainerStarted","Data":"cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f"} Oct 31 01:34:57.107005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:57.107023 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" event={"ID":"e88f9b32-4ce1-411c-a57e-f891e549b821","Type":"ContainerStarted","Data":"999abdbc6b866abccffc2d8c2034548ae0550728194eb4ac804fcebd0dcc8a1f"} Oct 31 01:34:57.107254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:57.107067 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:34:57.226583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:34:57.226463 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" podStartSLOduration=1.2264443 podStartE2EDuration="1.2264443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:34:57.225000258 +0000 UTC m=+2269.031967802" watchObservedRunningTime="2026-10-31 01:34:57.2264443 +0000 UTC m=+2269.033411843" Oct 31 01:35:02.066605 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:35:02.066568 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:35:04.494277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:04.494244 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:35:08.117308 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:08.117275 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:35:09.787655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:09.787619 5756 scope.go:122] "RemoveContainer" containerID="7e1ca0d48e3c865cee78f6ccb30aede49dcdad2856a640e30832f2751affeaa5" Oct 31 01:35:13.487510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.487469 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp"] Oct 31 01:35:13.493986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.493957 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.546215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.546180 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3f0818-openshift-default-dockercfg-r4qc6\"" Oct 31 01:35:13.598339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.598295 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp"] Oct 31 01:35:13.630163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630131 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630172 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggncb\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630196 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630448 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630507 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630589 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630628 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630674 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.630757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.630730 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732382 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732432 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggncb\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732461 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732604 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732637 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732662 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.732704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732702 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732774 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.732814 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.733131 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733308 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.733288 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733381 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.733307 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733381 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.733349 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.733611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.733590 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.735300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.735279 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.742921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.736315 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.789565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.789526 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.791203 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.791172 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggncb\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb\") pod \"e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:13.808263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:13.808002 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:14.252634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:14.252590 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp"] Oct 31 01:35:14.256993 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:35:14.256952 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70abbbf_469e_495d_bd1f_0101de39b653.slice/crio-14327b4cb1d6fd20d4fa0837f914482671a58c69aeed44eaa5f241151bc41a2a WatchSource:0}: Error finding container 14327b4cb1d6fd20d4fa0837f914482671a58c69aeed44eaa5f241151bc41a2a: Status 404 returned error can't find the container with id 14327b4cb1d6fd20d4fa0837f914482671a58c69aeed44eaa5f241151bc41a2a Oct 31 01:35:14.260199 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:14.260158 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:35:14.260313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:14.260247 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:35:14.260313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:14.260289 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:35:15.243569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:15.243532 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" event={"ID":"c70abbbf-469e-495d-bd1f-0101de39b653","Type":"ContainerStarted","Data":"118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d"} Oct 31 01:35:15.244065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:15.243585 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" event={"ID":"c70abbbf-469e-495d-bd1f-0101de39b653","Type":"ContainerStarted","Data":"14327b4cb1d6fd20d4fa0837f914482671a58c69aeed44eaa5f241151bc41a2a"} Oct 31 01:35:15.462692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:15.462635 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podStartSLOduration=2.46262123 podStartE2EDuration="2.46262123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:15.412384508 +0000 UTC m=+2287.219352052" watchObservedRunningTime="2026-10-31 01:35:15.46262123 +0000 UTC m=+2287.269588772" Oct 31 01:35:15.808190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:15.808151 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:15.814312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:15.814287 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:16.251162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:16.251075 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:16.252006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:16.251987 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:35:32.067427 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:35:32.067382 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:35:33.658956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:33.658915 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:35:41.049371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:41.049339 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj"] Oct 31 01:35:41.098788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:41.098755 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c8a90d2-w4qcj"] Oct 31 01:35:42.311984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:42.311946 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35c015a-efa9-436f-be81-6216594194ed" path="/var/lib/kubelet/pods/f35c015a-efa9-436f-be81-6216594194ed/volumes" Oct 31 01:35:48.505126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:48.505093 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz"] Oct 31 01:35:48.505613 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:48.505302 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" containerID="cri-o://086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6" gracePeriod=30 Oct 31 01:35:53.772459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.772404 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:35:53.926304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926273 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926323 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926367 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926394 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926445 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926471 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926574 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926665 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.926853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926715 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:35:53.926853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926738 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs\") pod \"c70c7134-e696-4718-b986-94ccb0d1f0bc\" (UID: \"c70c7134-e696-4718-b986-94ccb0d1f0bc\") " Oct 31 01:35:53.927006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926943 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:35:53.927006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926957 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:35:53.927006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.926856 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:35:53.927158 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.927134 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:35:53.927547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.927523 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:53.927654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.927554 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:53.927654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.927571 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:53.927654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.927586 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:53.927654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.927600 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70c7134-e696-4718-b986-94ccb0d1f0bc-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:53.928842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.928806 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:35:53.928972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.928845 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "kube-api-access-jwxgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:35:53.928972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.928915 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:35:53.929241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:53.929222 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token" pod "c70c7134-e696-4718-b986-94ccb0d1f0bc" (UID: "c70c7134-e696-4718-b986-94ccb0d1f0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:35:54.029309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.029232 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:54.029309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.029262 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:54.029309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.029271 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwxgw\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-kube-api-access-jwxgw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:54.029309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.029281 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70c7134-e696-4718-b986-94ccb0d1f0bc-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:35:54.487220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.487183 5756 generic.go:356] "Generic (PLEG): container finished" podID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerID="086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6" exitCode=0 Oct 31 01:35:54.487405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.487230 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" event={"ID":"c70c7134-e696-4718-b986-94ccb0d1f0bc","Type":"ContainerDied","Data":"086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6"} Oct 31 01:35:54.487405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.487247 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" Oct 31 01:35:54.487405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.487284 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz" event={"ID":"c70c7134-e696-4718-b986-94ccb0d1f0bc","Type":"ContainerDied","Data":"e46ef00a1942e09ed74ae35b1667ebacc04251d6366ae50f713f5ec76fb60ef8"} Oct 31 01:35:54.487405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.487313 5756 scope.go:122] "RemoveContainer" containerID="086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6" Oct 31 01:35:54.496992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.496971 5756 scope.go:122] "RemoveContainer" containerID="086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6" Oct 31 01:35:54.497256 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:35:54.497239 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6\": container with ID starting with 086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6 not found: ID does not exist" containerID="086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6" Oct 31 01:35:54.497338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.497264 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6"} err="failed to get container status \"086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6\": rpc error: code = NotFound desc = could not find container \"086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6\": container with ID starting with 086050710ad7a2360e4bf9d20d7f6aa239339923036652fc44344f4b4115e1b6 not found: ID does not exist" Oct 31 01:35:54.700635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.700554 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz"] Oct 31 01:35:54.750011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:54.749973 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-acd385-openshift-default-6f6f9f6f45-qqstz"] Oct 31 01:35:56.311787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:35:56.311753 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" path="/var/lib/kubelet/pods/c70c7134-e696-4718-b986-94ccb0d1f0bc/volumes" Oct 31 01:36:02.065368 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:36:02.065314 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:36:09.814711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:09.814677 5756 scope.go:122] "RemoveContainer" containerID="a8b91554d1dbda5ea93f0a7cc0f0b8dc863229727dd1c09c6e52c45d8f827c32" Oct 31 01:36:09.823770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:09.823749 5756 scope.go:122] "RemoveContainer" containerID="9f5002c0d2736c51189ffbe19410ec9898465154405d32e1bf13e6b3a16c60b3" Oct 31 01:36:16.778891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.778802 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk"] Oct 31 01:36:16.779435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.779419 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" Oct 31 01:36:16.779479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.779436 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" Oct 31 01:36:16.779543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.779531 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70c7134-e696-4718-b986-94ccb0d1f0bc" containerName="istio-proxy" Oct 31 01:36:16.784166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.784140 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.837205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.837176 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:36:16.885977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.885941 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:36:16.888375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.888345 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk"] Oct 31 01:36:16.895435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.895411 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lsk9\" (UniqueName: \"kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.895598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.895579 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.895671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.895656 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.996664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.996626 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.996898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.996749 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.996898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.996820 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lsk9\" (UniqueName: \"kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.997303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.997278 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:16.999220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:16.999197 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:17.063389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:17.063357 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lsk9\" (UniqueName: \"kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9\") pod \"maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:17.098046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:17.097993 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:17.499154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:17.499080 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk"] Oct 31 01:36:17.503814 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:36:17.503771 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4e3e2d_a6e8_45bf_8d5c_e9f85b04ee54.slice/crio-6c0cff7d628396901283cc1f6ec7e0dd2fc73853418e7c8eb32f8f9f4ae114d4 WatchSource:0}: Error finding container 6c0cff7d628396901283cc1f6ec7e0dd2fc73853418e7c8eb32f8f9f4ae114d4: Status 404 returned error can't find the container with id 6c0cff7d628396901283cc1f6ec7e0dd2fc73853418e7c8eb32f8f9f4ae114d4 Oct 31 01:36:17.628555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:17.628513 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" event={"ID":"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54","Type":"ContainerStarted","Data":"6c0cff7d628396901283cc1f6ec7e0dd2fc73853418e7c8eb32f8f9f4ae114d4"} Oct 31 01:36:18.637156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:18.637120 5756 generic.go:356] "Generic (PLEG): container finished" podID="6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" containerID="dce1c52722cd2f862bd6e06663c54a79cdb2ad77fdfbe61fc00ce3961d6ed413" exitCode=0 Oct 31 01:36:18.637557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:18.637222 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" event={"ID":"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54","Type":"ContainerDied","Data":"dce1c52722cd2f862bd6e06663c54a79cdb2ad77fdfbe61fc00ce3961d6ed413"} Oct 31 01:36:19.796645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.796622 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:19.931887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.931781 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca\") pod \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " Oct 31 01:36:19.932086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.931902 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca\") pod \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " Oct 31 01:36:19.932086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.931963 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9\" (UniqueName: \"kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9\") pod \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\" (UID: \"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54\") " Oct 31 01:36:19.932376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.932351 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca" pod "6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" (UID: "6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:36:19.934124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.934097 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca" pod "6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" (UID: "6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:19.934255 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:19.934239 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9" pod "6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" (UID: "6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54"). InnerVolumeSpecName "kube-api-access-6lsk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:20.033841 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:20.033810 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:36:20.033841 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:20.033844 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:36:20.034070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:20.033875 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lsk9\" (UniqueName: \"kubernetes.io/projected/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54-kube-api-access-6lsk9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:36:20.650017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:20.649940 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" Oct 31 01:36:20.650017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:20.649965 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk" event={"ID":"6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54","Type":"ContainerDied","Data":"6c0cff7d628396901283cc1f6ec7e0dd2fc73853418e7c8eb32f8f9f4ae114d4"} Oct 31 01:36:20.650017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:20.650015 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0cff7d628396901283cc1f6ec7e0dd2fc73853418e7c8eb32f8f9f4ae114d4" Oct 31 01:36:30.754766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:30.754730 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:36:31.105594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.105550 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5"] Oct 31 01:36:31.106190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.106169 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" containerName="revoke-keys" Oct 31 01:36:31.106316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.106191 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" containerName="revoke-keys" Oct 31 01:36:31.106414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.106399 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" containerName="revoke-keys" Oct 31 01:36:31.109602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.109579 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.161385 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.161341 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.161573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.161508 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8444\" (UniqueName: \"kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444\") pod \"payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.213733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.213693 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6c7db9-dockercfg-mqlr8\"" Oct 31 01:36:31.213733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.213718 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6c7db9\"" Oct 31 01:36:31.215928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.215908 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5"] Oct 31 01:36:31.262610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.262571 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8444\" (UniqueName: \"kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444\") pod \"payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.262808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.262701 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.263611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.263589 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.345483 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.344680 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8444\" (UniqueName: \"kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444\") pod \"payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.423946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.423830 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:31.803691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.803668 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5"] Oct 31 01:36:31.806160 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:36:31.806134 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5357c559_a6a7_419a_be4e_b26ef903c71a.slice/crio-9bffeb05aa89662be60425d2943690f6a36998c5aac6a97a7a8f1ed9da108367 WatchSource:0}: Error finding container 9bffeb05aa89662be60425d2943690f6a36998c5aac6a97a7a8f1ed9da108367: Status 404 returned error can't find the container with id 9bffeb05aa89662be60425d2943690f6a36998c5aac6a97a7a8f1ed9da108367 Oct 31 01:36:31.920612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.920568 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479"] Oct 31 01:36:31.926084 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.926051 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:31.968377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.968303 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:31.968534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.968425 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trb75\" (UniqueName: \"kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75\") pod \"payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:31.977399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:31.977371 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479"] Oct 31 01:36:32.066932 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:36:32.066898 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:36:32.069563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.069540 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:32.069667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.069604 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trb75\" (UniqueName: \"kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75\") pod \"payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:32.070290 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.070268 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:32.126053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.126019 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trb75\" (UniqueName: \"kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75\") pod \"payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:32.243772 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.243691 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:32.615751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.615723 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479"] Oct 31 01:36:32.617906 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:36:32.617836 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e86551_ebed_4a03_8993_5742791926c7.slice/crio-6bc6ed9fb285c31aeb84670cb7f5032916a2c3a2e88e096f8bf34618e4b8473e WatchSource:0}: Error finding container 6bc6ed9fb285c31aeb84670cb7f5032916a2c3a2e88e096f8bf34618e4b8473e: Status 404 returned error can't find the container with id 6bc6ed9fb285c31aeb84670cb7f5032916a2c3a2e88e096f8bf34618e4b8473e Oct 31 01:36:32.721145 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.721107 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" event={"ID":"a4e86551-ebed-4a03-8993-5742791926c7","Type":"ContainerStarted","Data":"206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7"} Oct 31 01:36:32.721304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.721173 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" event={"ID":"a4e86551-ebed-4a03-8993-5742791926c7","Type":"ContainerStarted","Data":"6bc6ed9fb285c31aeb84670cb7f5032916a2c3a2e88e096f8bf34618e4b8473e"} Oct 31 01:36:32.721304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.721200 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:32.722938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.722906 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" event={"ID":"5357c559-a6a7-419a-be4e-b26ef903c71a","Type":"ContainerStarted","Data":"5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c"} Oct 31 01:36:32.723318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.722960 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" event={"ID":"5357c559-a6a7-419a-be4e-b26ef903c71a","Type":"ContainerStarted","Data":"9bffeb05aa89662be60425d2943690f6a36998c5aac6a97a7a8f1ed9da108367"} Oct 31 01:36:32.723318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.723010 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:32.833614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.833563 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" podStartSLOduration=1.833549808 podStartE2EDuration="1.833549808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:32.832682669 +0000 UTC m=+2364.639650217" watchObservedRunningTime="2026-10-31 01:36:32.833549808 +0000 UTC m=+2364.640517344" Oct 31 01:36:32.948479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:32.948301 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" podStartSLOduration=1.948281362 podStartE2EDuration="1.948281362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:32.945021161 +0000 UTC m=+2364.751988705" watchObservedRunningTime="2026-10-31 01:36:32.948281362 +0000 UTC m=+2364.755248906" Oct 31 01:36:43.731103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:43.731070 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:36:43.731589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:43.731152 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:36:56.346320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.346290 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd"] Oct 31 01:36:56.346916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.346487 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" podUID="e5cd785d-35b0-4008-a5f4-eb64139aee68" containerName="payload-processing" containerID="cri-o://03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7" gracePeriod=30 Oct 31 01:36:56.628297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.628272 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:36:56.666933 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.666907 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume\") pod \"e5cd785d-35b0-4008-a5f4-eb64139aee68\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " Oct 31 01:36:56.667117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.667102 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r\" (UniqueName: \"kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r\") pod \"e5cd785d-35b0-4008-a5f4-eb64139aee68\" (UID: \"e5cd785d-35b0-4008-a5f4-eb64139aee68\") " Oct 31 01:36:56.667398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.667360 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume" pod "e5cd785d-35b0-4008-a5f4-eb64139aee68" (UID: "e5cd785d-35b0-4008-a5f4-eb64139aee68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:36:56.667661 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.667635 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd785d-35b0-4008-a5f4-eb64139aee68-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:36:56.669419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.669395 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r" pod "e5cd785d-35b0-4008-a5f4-eb64139aee68" (UID: "e5cd785d-35b0-4008-a5f4-eb64139aee68"). InnerVolumeSpecName "kube-api-access-mzf6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:56.768549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.768509 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzf6r\" (UniqueName: \"kubernetes.io/projected/e5cd785d-35b0-4008-a5f4-eb64139aee68-kube-api-access-mzf6r\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:36:56.905352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.905252 5756 generic.go:356] "Generic (PLEG): container finished" podID="e5cd785d-35b0-4008-a5f4-eb64139aee68" containerID="03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7" exitCode=0 Oct 31 01:36:56.905352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.905315 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" Oct 31 01:36:56.905352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.905335 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" event={"ID":"e5cd785d-35b0-4008-a5f4-eb64139aee68","Type":"ContainerDied","Data":"03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7"} Oct 31 01:36:56.905615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.905392 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd" event={"ID":"e5cd785d-35b0-4008-a5f4-eb64139aee68","Type":"ContainerDied","Data":"067610332ae07bf3c890ed61227c1fb3072a1c7bdf602c593839b55ee87414f9"} Oct 31 01:36:56.905615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.905417 5756 scope.go:122] "RemoveContainer" containerID="03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7" Oct 31 01:36:56.919058 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.919038 5756 scope.go:122] "RemoveContainer" containerID="03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7" Oct 31 01:36:56.919358 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:36:56.919329 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7\": container with ID starting with 03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7 not found: ID does not exist" containerID="03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7" Oct 31 01:36:56.919458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:56.919370 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7"} err="failed to get container status \"03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7\": rpc error: code = NotFound desc = could not find container \"03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7\": container with ID starting with 03494c92b4347ce31f8baa4c4dfe829832127dd74e62ec6729d21046cbaa8ee7 not found: ID does not exist" Oct 31 01:36:57.072349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:57.072309 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd"] Oct 31 01:36:57.078981 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:57.078955 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b95dca-7f4f7b97d6-p4jqd"] Oct 31 01:36:58.131858 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:58.131821 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk"] Oct 31 01:36:58.134078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:58.134057 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b95dca-xfgbk"] Oct 31 01:36:58.312324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:58.312292 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54" path="/var/lib/kubelet/pods/6b4e3e2d-a6e8-45bf-8d5c-e9f85b04ee54/volumes" Oct 31 01:36:58.312661 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:36:58.312648 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cd785d-35b0-4008-a5f4-eb64139aee68" path="/var/lib/kubelet/pods/e5cd785d-35b0-4008-a5f4-eb64139aee68/volumes" Oct 31 01:37:02.066361 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:37:02.066324 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:37:04.066081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:04.066048 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns"] Oct 31 01:37:04.066473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:04.066241 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" containerID="cri-o://33753ee81b2c1ae9a559e5b728c6bce279773d515e659593d890e49ab1b4bff6" gracePeriod=30 Oct 31 01:37:05.498473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.498435 5756 patch_prober.go:29] interesting pod/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:37:05.498853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.498528 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:37:05.807677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.807644 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8"] Oct 31 01:37:05.808246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.808230 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e5cd785d-35b0-4008-a5f4-eb64139aee68" containerName="payload-processing" Oct 31 01:37:05.808310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.808248 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cd785d-35b0-4008-a5f4-eb64139aee68" containerName="payload-processing" Oct 31 01:37:05.808384 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.808374 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cd785d-35b0-4008-a5f4-eb64139aee68" containerName="payload-processing" Oct 31 01:37:05.811841 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.811825 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.890013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.889971 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.890178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.890030 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.890248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.890221 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kn4\" (UniqueName: \"kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.914462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.914430 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:37:05.914628 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.914532 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:37:05.916829 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.916804 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8"] Oct 31 01:37:05.991135 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.991104 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.991307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.991148 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.991307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.991197 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kn4\" (UniqueName: \"kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.991960 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.991882 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:05.993663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:05.993638 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:06.047900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:06.047843 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kn4\" (UniqueName: \"kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4\") pod \"maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:06.125736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:06.125651 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:06.498220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:06.498190 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8"] Oct 31 01:37:06.500384 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:37:06.500355 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd868c2f6_3045_4f70_9f05_6540dd3b1a54.slice/crio-ac7a058ba79a5a8be2ded4e854fc946cd3ad8feb6bc0438cc9c5dd604127c384 WatchSource:0}: Error finding container ac7a058ba79a5a8be2ded4e854fc946cd3ad8feb6bc0438cc9c5dd604127c384: Status 404 returned error can't find the container with id ac7a058ba79a5a8be2ded4e854fc946cd3ad8feb6bc0438cc9c5dd604127c384 Oct 31 01:37:07.008736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:07.008642 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" event={"ID":"d868c2f6-3045-4f70-9f05-6540dd3b1a54","Type":"ContainerStarted","Data":"4ea5597cf2412766c3c3a78a4c46285c73a21b753d9fe4ebbcc10edf28fa8c42"} Oct 31 01:37:07.008923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:07.008736 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" event={"ID":"d868c2f6-3045-4f70-9f05-6540dd3b1a54","Type":"ContainerStarted","Data":"ac7a058ba79a5a8be2ded4e854fc946cd3ad8feb6bc0438cc9c5dd604127c384"} Oct 31 01:37:07.122419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:07.122363 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" podStartSLOduration=2.122348564 podStartE2EDuration="2.122348564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:07.122200702 +0000 UTC m=+2398.929168246" watchObservedRunningTime="2026-10-31 01:37:07.122348564 +0000 UTC m=+2398.929316112" Oct 31 01:37:08.016019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:08.015979 5756 generic.go:356] "Generic (PLEG): container finished" podID="d868c2f6-3045-4f70-9f05-6540dd3b1a54" containerID="4ea5597cf2412766c3c3a78a4c46285c73a21b753d9fe4ebbcc10edf28fa8c42" exitCode=0 Oct 31 01:37:08.016407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:08.016026 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" event={"ID":"d868c2f6-3045-4f70-9f05-6540dd3b1a54","Type":"ContainerDied","Data":"4ea5597cf2412766c3c3a78a4c46285c73a21b753d9fe4ebbcc10edf28fa8c42"} Oct 31 01:37:09.202992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.202640 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:09.331837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.331810 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4\" (UniqueName: \"kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4\") pod \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " Oct 31 01:37:09.332051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.331988 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca\") pod \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " Oct 31 01:37:09.332182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.332071 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca\") pod \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\" (UID: \"d868c2f6-3045-4f70-9f05-6540dd3b1a54\") " Oct 31 01:37:09.332476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.332452 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca" pod "d868c2f6-3045-4f70-9f05-6540dd3b1a54" (UID: "d868c2f6-3045-4f70-9f05-6540dd3b1a54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:09.334238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.334211 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4" pod "d868c2f6-3045-4f70-9f05-6540dd3b1a54" (UID: "d868c2f6-3045-4f70-9f05-6540dd3b1a54"). InnerVolumeSpecName "kube-api-access-65kn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:09.334329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.334276 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca" pod "d868c2f6-3045-4f70-9f05-6540dd3b1a54" (UID: "d868c2f6-3045-4f70-9f05-6540dd3b1a54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:09.335000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.334980 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:37:09.432789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.432753 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.432992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.432810 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.432992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.432951 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.432992 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433032 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433034 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:09.433459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433316 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:09.433459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433380 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433454 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433678 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433498 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433678 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433536 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket\") pod \"2f82f9f2-6da1-495b-98e5-9c00182e6691\" (UID: \"2f82f9f2-6da1-495b-98e5-9c00182e6691\") " Oct 31 01:37:09.433813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433792 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:09.433921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.433818 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:09.434027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434002 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:09.434400 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434378 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434408 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434425 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f82f9f2-6da1-495b-98e5-9c00182e6691-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434439 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434454 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434469 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d868c2f6-3045-4f70-9f05-6540dd3b1a54-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434485 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65kn4\" (UniqueName: \"kubernetes.io/projected/d868c2f6-3045-4f70-9f05-6540dd3b1a54-kube-api-access-65kn4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.434668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.434499 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.435900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.435839 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:09.436030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.435895 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "kube-api-access-hmcxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:09.436030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.435973 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:09.437223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.437202 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo" pod "2f82f9f2-6da1-495b-98e5-9c00182e6691" (UID: "2f82f9f2-6da1-495b-98e5-9c00182e6691"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:37:09.535612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.535582 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.535612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.535610 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.535612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.535620 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f82f9f2-6da1-495b-98e5-9c00182e6691-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.535849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.535632 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmcxh\" (UniqueName: \"kubernetes.io/projected/2f82f9f2-6da1-495b-98e5-9c00182e6691-kube-api-access-hmcxh\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:09.860113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.860078 5756 scope.go:122] "RemoveContainer" containerID="af5278d7b78442ece74fbfc381b857095d838bb49a9536dc924e86d6d4331cb4" Oct 31 01:37:09.869897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.869854 5756 scope.go:122] "RemoveContainer" containerID="33753ee81b2c1ae9a559e5b728c6bce279773d515e659593d890e49ab1b4bff6" Oct 31 01:37:09.878588 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.878566 5756 scope.go:122] "RemoveContainer" containerID="af31e4dd8f375370fa38c60d3c92fd66967dc68de05961ff2600d2a37cde6114" Oct 31 01:37:09.887069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:09.887048 5756 scope.go:122] "RemoveContainer" containerID="916cc2ce6e3ecf5a79ad018a7321466143d7cbb1c759cd1e829cdec8f57ae417" Oct 31 01:37:10.029640 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.029598 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" event={"ID":"2f82f9f2-6da1-495b-98e5-9c00182e6691","Type":"ContainerDied","Data":"33753ee81b2c1ae9a559e5b728c6bce279773d515e659593d890e49ab1b4bff6"} Oct 31 01:37:10.029838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.029674 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" event={"ID":"2f82f9f2-6da1-495b-98e5-9c00182e6691","Type":"ContainerDied","Data":"676ecbfed86af80d5ed837a28de3c3461fa5ef57991aea6260c13f8ecd93d8bc"} Oct 31 01:37:10.031448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.031425 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns" Oct 31 01:37:10.031969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.031944 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" event={"ID":"d868c2f6-3045-4f70-9f05-6540dd3b1a54","Type":"ContainerDied","Data":"ac7a058ba79a5a8be2ded4e854fc946cd3ad8feb6bc0438cc9c5dd604127c384"} Oct 31 01:37:10.032076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.031997 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7a058ba79a5a8be2ded4e854fc946cd3ad8feb6bc0438cc9c5dd604127c384" Oct 31 01:37:10.032139 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.032077 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8" Oct 31 01:37:10.318915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.318852 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns"] Oct 31 01:37:10.325192 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:10.325164 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ec695b-openshift-default-5d78d64df9-c8xns"] Oct 31 01:37:12.311623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:12.311588 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" path="/var/lib/kubelet/pods/2f82f9f2-6da1-495b-98e5-9c00182e6691/volumes" Oct 31 01:37:16.933294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:16.933256 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:37:26.291784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:26.291752 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4"] Oct 31 01:37:26.292210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:26.292010 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" containerID="cri-o://2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67" gracePeriod=30 Oct 31 01:37:26.470959 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:26.470914 5756 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fc8c56547-6xjn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:37:26.471121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:26.470993 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:37:27.127920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:27.127886 5756 generic.go:356] "Generic (PLEG): container finished" podID="c70abbbf-469e-495d-bd1f-0101de39b653" containerID="118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d" exitCode=0 Oct 31 01:37:27.128099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:27.127930 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" event={"ID":"c70abbbf-469e-495d-bd1f-0101de39b653","Type":"ContainerDied","Data":"118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d"} Oct 31 01:37:27.128335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:27.128322 5756 scope.go:122] "RemoveContainer" containerID="118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d" Oct 31 01:37:27.129842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:27.129810 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:37:27.129995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:27.129914 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:37:27.129995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:27.129962 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:37:28.154918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:28.154084 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" event={"ID":"c70abbbf-469e-495d-bd1f-0101de39b653","Type":"ContainerStarted","Data":"48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d"} Oct 31 01:37:28.808760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:28.808727 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:37:28.810164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:28.810141 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:37:28.810337 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:28.810313 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 01:37:29.808368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:29.808332 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:37:29.808889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:29.808386 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 01:37:30.808416 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:30.808384 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:37:30.808913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:30.808855 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 01:37:31.251052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.250960 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:37:31.591680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.591654 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:37:31.721826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.721800 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.721854 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.721920 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.721969 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722014 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722086 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722134 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722198 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:31.722258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722245 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722285 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert\") pod \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\" (UID: \"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d\") " Oct 31 01:37:31.722481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722348 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:31.722601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722481 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:31.722702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722566 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:31.722815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722718 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.722815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722777 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.722815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.722799 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.723212 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.723187 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:31.724230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.724206 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:31.724669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.724640 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:37:31.724877 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.724840 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "kube-api-access-4tm95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:31.725173 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.725154 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy" pod "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" (UID: "5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:37:31.808533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.808227 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:37:31.808533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.808280 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 01:37:31.824730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.824646 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.824928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.824829 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tm95\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-kube-api-access-4tm95\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.824928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.824852 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.824928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.824881 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.825516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.825493 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:31.825552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:31.825525 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:32.081893 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:37:32.081792 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:37:32.177449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.177407 5756 generic.go:356] "Generic (PLEG): container finished" podID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerID="2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67" exitCode=0 Oct 31 01:37:32.177636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.177525 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" event={"ID":"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d","Type":"ContainerDied","Data":"2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67"} Oct 31 01:37:32.177636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.177576 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" event={"ID":"5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d","Type":"ContainerDied","Data":"3d462775affdf510d172457d5889af8ccae060ef98f7386359cb3f96aa4c8fb8"} Oct 31 01:37:32.177636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.177606 5756 scope.go:122] "RemoveContainer" containerID="2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67" Oct 31 01:37:32.177813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.177763 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4" Oct 31 01:37:32.189247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.189227 5756 scope.go:122] "RemoveContainer" containerID="2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67" Oct 31 01:37:32.189507 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:37:32.189490 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67\": container with ID starting with 2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67 not found: ID does not exist" containerID="2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67" Oct 31 01:37:32.189562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.189516 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67"} err="failed to get container status \"2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67\": rpc error: code = NotFound desc = could not find container \"2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67\": container with ID starting with 2b7c5ef91959eb6341fe626613cf2074d2acd129255d5a9ca10883f32acafd67 not found: ID does not exist" Oct 31 01:37:32.257645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.257615 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc"] Oct 31 01:37:32.258172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258143 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d868c2f6-3045-4f70-9f05-6540dd3b1a54" containerName="revoke-keys" Oct 31 01:37:32.258172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258164 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d868c2f6-3045-4f70-9f05-6540dd3b1a54" containerName="revoke-keys" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258193 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258198 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258223 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258229 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258301 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f82f9f2-6da1-495b-98e5-9c00182e6691" containerName="istio-proxy" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258314 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" containerName="istio-proxy" Oct 31 01:37:32.258342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.258322 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d868c2f6-3045-4f70-9f05-6540dd3b1a54" containerName="revoke-keys" Oct 31 01:37:32.263018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.262997 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.314429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.314402 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:37:32.314611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.314590 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:37:32.358168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.358065 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrvt\" (UniqueName: \"kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.358168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.358158 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.358413 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.358190 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.363624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.363587 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc"] Oct 31 01:37:32.368067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.367311 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4"] Oct 31 01:37:32.371150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.369579 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fc8c56547-6xjn4"] Oct 31 01:37:32.460935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.460850 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrvt\" (UniqueName: \"kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.461151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.460995 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.461151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.461035 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.461904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.461878 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.464760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.464712 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.517923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.517855 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrvt\" (UniqueName: \"kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt\") pod \"maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.577481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.577436 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:32.808392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.808357 5756 patch_prober.go:29] interesting pod/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:37:32.808561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.808423 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 31 01:37:32.975441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:32.975413 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc"] Oct 31 01:37:32.977961 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:37:32.977929 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3ad52c_7567_43f4_88d3_514012a4bc70.slice/crio-e49a86979ca6a45ebfd33552e6fdbf524b6c840560322e748fcfdc163fb14ac2 WatchSource:0}: Error finding container e49a86979ca6a45ebfd33552e6fdbf524b6c840560322e748fcfdc163fb14ac2: Status 404 returned error can't find the container with id e49a86979ca6a45ebfd33552e6fdbf524b6c840560322e748fcfdc163fb14ac2 Oct 31 01:37:33.186487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:33.186445 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" event={"ID":"6f3ad52c-7567-43f4-88d3-514012a4bc70","Type":"ContainerStarted","Data":"e49a86979ca6a45ebfd33552e6fdbf524b6c840560322e748fcfdc163fb14ac2"} Oct 31 01:37:33.813040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:33.813013 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:37:33.814021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:33.814003 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:37:34.193363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:34.193281 5756 generic.go:356] "Generic (PLEG): container finished" podID="6f3ad52c-7567-43f4-88d3-514012a4bc70" containerID="62ce6f80c9c3e3c63f8e36150cdc625057e1cafe8e6f9e5f7e2809f5b5a9aba2" exitCode=0 Oct 31 01:37:34.193528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:34.193360 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" event={"ID":"6f3ad52c-7567-43f4-88d3-514012a4bc70","Type":"ContainerDied","Data":"62ce6f80c9c3e3c63f8e36150cdc625057e1cafe8e6f9e5f7e2809f5b5a9aba2"} Oct 31 01:37:34.313817 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:34.313788 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d" path="/var/lib/kubelet/pods/5d8c6f08-8b82-4bc9-8cdf-9a23f066ec1d/volumes" Oct 31 01:37:34.487857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:34.487779 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8"] Oct 31 01:37:34.489833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:34.489810 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c0359f-kpfr8"] Oct 31 01:37:35.353210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.353184 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:35.503007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.502928 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt\" (UniqueName: \"kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt\") pod \"6f3ad52c-7567-43f4-88d3-514012a4bc70\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " Oct 31 01:37:35.503162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.503124 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca\") pod \"6f3ad52c-7567-43f4-88d3-514012a4bc70\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " Oct 31 01:37:35.503202 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.503170 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca\") pod \"6f3ad52c-7567-43f4-88d3-514012a4bc70\" (UID: \"6f3ad52c-7567-43f4-88d3-514012a4bc70\") " Oct 31 01:37:35.503496 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.503456 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca" pod "6f3ad52c-7567-43f4-88d3-514012a4bc70" (UID: "6f3ad52c-7567-43f4-88d3-514012a4bc70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:35.505279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.505255 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt" pod "6f3ad52c-7567-43f4-88d3-514012a4bc70" (UID: "6f3ad52c-7567-43f4-88d3-514012a4bc70"). InnerVolumeSpecName "kube-api-access-nbrvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:35.505373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.505321 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca" pod "6f3ad52c-7567-43f4-88d3-514012a4bc70" (UID: "6f3ad52c-7567-43f4-88d3-514012a4bc70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:35.604762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.604712 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:35.604762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.604757 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f3ad52c-7567-43f4-88d3-514012a4bc70-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:35.604762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:35.604766 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrvt\" (UniqueName: \"kubernetes.io/projected/6f3ad52c-7567-43f4-88d3-514012a4bc70-kube-api-access-nbrvt\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:36.205281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:36.205236 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" event={"ID":"6f3ad52c-7567-43f4-88d3-514012a4bc70","Type":"ContainerDied","Data":"e49a86979ca6a45ebfd33552e6fdbf524b6c840560322e748fcfdc163fb14ac2"} Oct 31 01:37:36.205452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:36.205297 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49a86979ca6a45ebfd33552e6fdbf524b6c840560322e748fcfdc163fb14ac2" Oct 31 01:37:36.205452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:36.205249 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc" Oct 31 01:37:36.311695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:36.311662 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d868c2f6-3045-4f70-9f05-6540dd3b1a54" path="/var/lib/kubelet/pods/d868c2f6-3045-4f70-9f05-6540dd3b1a54/volumes" Oct 31 01:37:43.220517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:43.220477 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb"] Oct 31 01:37:48.221740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.221644 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" podUID="e88f9b32-4ce1-411c-a57e-f891e549b821" containerName="payload-processing" containerID="cri-o://cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f" gracePeriod=25 Oct 31 01:37:48.495254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.495226 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:37:48.558098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.558064 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume\") pod \"e88f9b32-4ce1-411c-a57e-f891e549b821\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " Oct 31 01:37:48.558251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.558108 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59\" (UniqueName: \"kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59\") pod \"e88f9b32-4ce1-411c-a57e-f891e549b821\" (UID: \"e88f9b32-4ce1-411c-a57e-f891e549b821\") " Oct 31 01:37:48.558431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.558410 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume" pod "e88f9b32-4ce1-411c-a57e-f891e549b821" (UID: "e88f9b32-4ce1-411c-a57e-f891e549b821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:48.560375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.560349 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59" pod "e88f9b32-4ce1-411c-a57e-f891e549b821" (UID: "e88f9b32-4ce1-411c-a57e-f891e549b821"). InnerVolumeSpecName "kube-api-access-2dq59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:48.660311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.660276 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e88f9b32-4ce1-411c-a57e-f891e549b821-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:48.660493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:48.660321 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dq59\" (UniqueName: \"kubernetes.io/projected/e88f9b32-4ce1-411c-a57e-f891e549b821-kube-api-access-2dq59\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:37:49.283630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.283593 5756 generic.go:356] "Generic (PLEG): container finished" podID="e88f9b32-4ce1-411c-a57e-f891e549b821" containerID="cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f" exitCode=0 Oct 31 01:37:49.284128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.283657 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" Oct 31 01:37:49.284128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.283651 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" event={"ID":"e88f9b32-4ce1-411c-a57e-f891e549b821","Type":"ContainerDied","Data":"cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f"} Oct 31 01:37:49.284128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.283811 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb" event={"ID":"e88f9b32-4ce1-411c-a57e-f891e549b821","Type":"ContainerDied","Data":"999abdbc6b866abccffc2d8c2034548ae0550728194eb4ac804fcebd0dcc8a1f"} Oct 31 01:37:49.284128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.283838 5756 scope.go:122] "RemoveContainer" containerID="cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f" Oct 31 01:37:49.295094 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.295071 5756 scope.go:122] "RemoveContainer" containerID="cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f" Oct 31 01:37:49.295393 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:37:49.295364 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f\": container with ID starting with cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f not found: ID does not exist" containerID="cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f" Oct 31 01:37:49.295481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.295403 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f"} err="failed to get container status \"cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f\": rpc error: code = NotFound desc = could not find container \"cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f\": container with ID starting with cdff7817b484d83ee6af593ba2d1f7fe885aa206603550c85970a435c30aa51f not found: ID does not exist" Oct 31 01:37:49.445424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.445346 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb"] Oct 31 01:37:49.495290 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:49.495259 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a31828-7c64bcbb89-nn7fb"] Oct 31 01:37:50.311891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:50.311834 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88f9b32-4ce1-411c-a57e-f891e549b821" path="/var/lib/kubelet/pods/e88f9b32-4ce1-411c-a57e-f891e549b821/volumes" Oct 31 01:37:58.957963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.957926 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4"] Oct 31 01:37:58.958679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.958656 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e88f9b32-4ce1-411c-a57e-f891e549b821" containerName="payload-processing" Oct 31 01:37:58.958779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.958682 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88f9b32-4ce1-411c-a57e-f891e549b821" containerName="payload-processing" Oct 31 01:37:58.958779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.958728 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6f3ad52c-7567-43f4-88d3-514012a4bc70" containerName="revoke-keys" Oct 31 01:37:58.958779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.958736 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f3ad52c-7567-43f4-88d3-514012a4bc70" containerName="revoke-keys" Oct 31 01:37:58.958956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.958916 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88f9b32-4ce1-411c-a57e-f891e549b821" containerName="payload-processing" Oct 31 01:37:58.958956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.958939 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f3ad52c-7567-43f4-88d3-514012a4bc70" containerName="revoke-keys" Oct 31 01:37:58.963543 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:58.963523 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.064706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.064677 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-634135\"" Oct 31 01:37:59.064706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.064685 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-634135-dockercfg-pcw2t\"" Oct 31 01:37:59.066913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.066890 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4"] Oct 31 01:37:59.091417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.091391 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxkpq\" (UniqueName: \"kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq\") pod \"payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.091561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.091448 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.192421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.192381 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxkpq\" (UniqueName: \"kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq\") pod \"payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.192644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.192489 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.193256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.193232 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.248697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.248622 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxkpq\" (UniqueName: \"kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq\") pod \"payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.276904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.276855 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:37:59.654039 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:37:59.654016 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4"] Oct 31 01:37:59.656305 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:37:59.656277 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99a3ee11_a307_49b7_b46f_a4ada82148bd.slice/crio-31c0e84524ab4523f0338502b90bc709a93abe638f5a9c7c32bc4709b8fbea72 WatchSource:0}: Error finding container 31c0e84524ab4523f0338502b90bc709a93abe638f5a9c7c32bc4709b8fbea72: Status 404 returned error can't find the container with id 31c0e84524ab4523f0338502b90bc709a93abe638f5a9c7c32bc4709b8fbea72 Oct 31 01:38:00.368893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:00.368838 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" event={"ID":"99a3ee11-a307-49b7-b46f-a4ada82148bd","Type":"ContainerStarted","Data":"eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db"} Oct 31 01:38:00.368893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:00.368904 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" event={"ID":"99a3ee11-a307-49b7-b46f-a4ada82148bd","Type":"ContainerStarted","Data":"31c0e84524ab4523f0338502b90bc709a93abe638f5a9c7c32bc4709b8fbea72"} Oct 31 01:38:00.369373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:00.369057 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:38:00.483525 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:00.483472 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" podStartSLOduration=2.483456316 podStartE2EDuration="2.483456316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:00.481183362 +0000 UTC m=+2452.288150905" watchObservedRunningTime="2026-10-31 01:38:00.483456316 +0000 UTC m=+2452.290423858" Oct 31 01:38:02.071488 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:02.071444 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:38:02.201643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.201600 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv"] Oct 31 01:38:02.205412 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.205394 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.227254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.227215 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jfb\" (UniqueName: \"kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb\") pod \"payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.227387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.227264 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.262338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.262308 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv"] Oct 31 01:38:02.308677 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.308650 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6a9710-dockercfg-sf8lr\"" Oct 31 01:38:02.308843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.308682 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6a9710\"" Oct 31 01:38:02.328750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.328666 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jfb\" (UniqueName: \"kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb\") pod \"payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.328750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.328728 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.329453 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.329433 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.385424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.385393 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jfb\" (UniqueName: \"kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb\") pod \"payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.519920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.519856 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:02.893951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:02.893926 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv"] Oct 31 01:38:02.896629 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:38:02.896600 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362d427b_b6d5_457b_9a01_0648274ff655.slice/crio-e40902da4544583b0a39c4511570c853ef18e85ed89dd77fcd2c0ee5c7469296 WatchSource:0}: Error finding container e40902da4544583b0a39c4511570c853ef18e85ed89dd77fcd2c0ee5c7469296: Status 404 returned error can't find the container with id e40902da4544583b0a39c4511570c853ef18e85ed89dd77fcd2c0ee5c7469296 Oct 31 01:38:03.388665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:03.388626 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" event={"ID":"362d427b-b6d5-457b-9a01-0648274ff655","Type":"ContainerStarted","Data":"37ca05d0abed4378a8593907521af437816b2ba94d9cd36fa2a57ee02e669443"} Oct 31 01:38:03.389107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:03.388684 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" event={"ID":"362d427b-b6d5-457b-9a01-0648274ff655","Type":"ContainerStarted","Data":"e40902da4544583b0a39c4511570c853ef18e85ed89dd77fcd2c0ee5c7469296"} Oct 31 01:38:03.389107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:03.388723 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:11.376827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:11.376795 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:38:11.492520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:11.492464 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" podStartSLOduration=9.492449409 podStartE2EDuration="9.492449409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:03.503971399 +0000 UTC m=+2455.310938943" watchObservedRunningTime="2026-10-31 01:38:11.492449409 +0000 UTC m=+2463.299416982" Oct 31 01:38:14.355176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:14.355145 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:38:14.399009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:14.398980 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:38:15.194649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.194616 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz"] Oct 31 01:38:15.199575 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.199556 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.274785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.274744 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.274785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.274784 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd97c\" (UniqueName: \"kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.275060 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.275019 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.302805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.302767 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:38:15.302805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.302767 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:38:15.306203 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.306182 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz"] Oct 31 01:38:15.375703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.375677 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.376166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.375749 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.376166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.375783 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd97c\" (UniqueName: \"kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.376511 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.376492 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.378247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.378222 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.432989 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.432965 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd97c\" (UniqueName: \"kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c\") pod \"maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.513069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.512983 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:15.895368 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:15.895336 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz"] Oct 31 01:38:15.899342 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:38:15.899307 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c0aec2_383d_4831_b8fa_a8a52ff4a4da.slice/crio-c7535387da51aa0e796c9b30ea963b0235771a6ed5b46a01450ef40f9417f482 WatchSource:0}: Error finding container c7535387da51aa0e796c9b30ea963b0235771a6ed5b46a01450ef40f9417f482: Status 404 returned error can't find the container with id c7535387da51aa0e796c9b30ea963b0235771a6ed5b46a01450ef40f9417f482 Oct 31 01:38:16.477679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:16.477642 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" event={"ID":"51c0aec2-383d-4831-b8fa-a8a52ff4a4da","Type":"ContainerStarted","Data":"a2d14b4ff755d2995b27b7a24db65434303cd7d9f7275e5363880476466afb1e"} Oct 31 01:38:16.478096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:16.477708 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" event={"ID":"51c0aec2-383d-4831-b8fa-a8a52ff4a4da","Type":"ContainerStarted","Data":"c7535387da51aa0e796c9b30ea963b0235771a6ed5b46a01450ef40f9417f482"} Oct 31 01:38:16.598260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:16.598206 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" podStartSLOduration=1.598192587 podStartE2EDuration="1.598192587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:16.597278759 +0000 UTC m=+2468.404246301" watchObservedRunningTime="2026-10-31 01:38:16.598192587 +0000 UTC m=+2468.405160171" Oct 31 01:38:17.485055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:17.485021 5756 generic.go:356] "Generic (PLEG): container finished" podID="51c0aec2-383d-4831-b8fa-a8a52ff4a4da" containerID="a2d14b4ff755d2995b27b7a24db65434303cd7d9f7275e5363880476466afb1e" exitCode=0 Oct 31 01:38:17.485454 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:17.485097 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" event={"ID":"51c0aec2-383d-4831-b8fa-a8a52ff4a4da","Type":"ContainerDied","Data":"a2d14b4ff755d2995b27b7a24db65434303cd7d9f7275e5363880476466afb1e"} Oct 31 01:38:18.640689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.640662 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:18.714094 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.714059 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca\") pod \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " Oct 31 01:38:18.714270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.714161 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca\") pod \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " Oct 31 01:38:18.714270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.714226 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c\" (UniqueName: \"kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c\") pod \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\" (UID: \"51c0aec2-383d-4831-b8fa-a8a52ff4a4da\") " Oct 31 01:38:18.714477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.714444 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca" pod "51c0aec2-383d-4831-b8fa-a8a52ff4a4da" (UID: "51c0aec2-383d-4831-b8fa-a8a52ff4a4da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:18.714690 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.714673 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:18.716716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.716689 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca" pod "51c0aec2-383d-4831-b8fa-a8a52ff4a4da" (UID: "51c0aec2-383d-4831-b8fa-a8a52ff4a4da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:38:18.716847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.716818 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c" pod "51c0aec2-383d-4831-b8fa-a8a52ff4a4da" (UID: "51c0aec2-383d-4831-b8fa-a8a52ff4a4da"). InnerVolumeSpecName "kube-api-access-dd97c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:18.816211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.816180 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:18.816211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:18.816211 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd97c\" (UniqueName: \"kubernetes.io/projected/51c0aec2-383d-4831-b8fa-a8a52ff4a4da-kube-api-access-dd97c\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:19.497130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:19.497100 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" Oct 31 01:38:19.497318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:19.497138 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz" event={"ID":"51c0aec2-383d-4831-b8fa-a8a52ff4a4da","Type":"ContainerDied","Data":"c7535387da51aa0e796c9b30ea963b0235771a6ed5b46a01450ef40f9417f482"} Oct 31 01:38:19.497318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:19.497192 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7535387da51aa0e796c9b30ea963b0235771a6ed5b46a01450ef40f9417f482" Oct 31 01:38:23.558499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.558462 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j"] Oct 31 01:38:23.559113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.559033 5756 cpu_manager.go:414] "containerMap: removing container" podUID="51c0aec2-383d-4831-b8fa-a8a52ff4a4da" containerName="revoke-keys" Oct 31 01:38:23.559113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.559046 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c0aec2-383d-4831-b8fa-a8a52ff4a4da" containerName="revoke-keys" Oct 31 01:38:23.559230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.559133 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c0aec2-383d-4831-b8fa-a8a52ff4a4da" containerName="revoke-keys" Oct 31 01:38:23.562835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.562806 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.617182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.617154 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:38:23.666568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.666530 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:38:23.669040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.669011 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j"] Oct 31 01:38:23.672277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.672246 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.672387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.672306 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.672442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.672428 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nqf\" (UniqueName: \"kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.773562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.773533 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.773763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.773589 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.773763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.773659 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nqf\" (UniqueName: \"kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.774273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.774244 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.776171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.776147 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.831421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.831342 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nqf\" (UniqueName: \"kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf\") pod \"maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:23.881172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:23.881142 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:24.265260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:24.265187 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j"] Oct 31 01:38:24.269566 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:38:24.269538 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb357ce02_37e1_4fc3_bc98_ed990300f797.slice/crio-d77d3cf99c7259d0fe3f59d8b5b7420e51afe73a1df9739098940cdf11f9a1ab WatchSource:0}: Error finding container d77d3cf99c7259d0fe3f59d8b5b7420e51afe73a1df9739098940cdf11f9a1ab: Status 404 returned error can't find the container with id d77d3cf99c7259d0fe3f59d8b5b7420e51afe73a1df9739098940cdf11f9a1ab Oct 31 01:38:24.525517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:24.525435 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" event={"ID":"b357ce02-37e1-4fc3-bc98-ed990300f797","Type":"ContainerStarted","Data":"d77d3cf99c7259d0fe3f59d8b5b7420e51afe73a1df9739098940cdf11f9a1ab"} Oct 31 01:38:25.534759 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:25.534721 5756 generic.go:356] "Generic (PLEG): container finished" podID="b357ce02-37e1-4fc3-bc98-ed990300f797" containerID="611046d1df284604b6b56438b04760bacf3388b2125be3779535f5187a2ec589" exitCode=0 Oct 31 01:38:25.535267 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:25.534766 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" event={"ID":"b357ce02-37e1-4fc3-bc98-ed990300f797","Type":"ContainerDied","Data":"611046d1df284604b6b56438b04760bacf3388b2125be3779535f5187a2ec589"} Oct 31 01:38:26.692750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.692725 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:26.809067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.808995 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca\") pod \"b357ce02-37e1-4fc3-bc98-ed990300f797\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " Oct 31 01:38:26.809067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.809043 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf\" (UniqueName: \"kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf\") pod \"b357ce02-37e1-4fc3-bc98-ed990300f797\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " Oct 31 01:38:26.809257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.809237 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca\") pod \"b357ce02-37e1-4fc3-bc98-ed990300f797\" (UID: \"b357ce02-37e1-4fc3-bc98-ed990300f797\") " Oct 31 01:38:26.809415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.809393 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca" pod "b357ce02-37e1-4fc3-bc98-ed990300f797" (UID: "b357ce02-37e1-4fc3-bc98-ed990300f797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:26.809673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.809658 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:26.811245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.811215 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf" pod "b357ce02-37e1-4fc3-bc98-ed990300f797" (UID: "b357ce02-37e1-4fc3-bc98-ed990300f797"). InnerVolumeSpecName "kube-api-access-d2nqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:26.811245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.811215 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca" pod "b357ce02-37e1-4fc3-bc98-ed990300f797" (UID: "b357ce02-37e1-4fc3-bc98-ed990300f797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:38:26.910667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.910631 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b357ce02-37e1-4fc3-bc98-ed990300f797-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:26.910667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:26.910662 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nqf\" (UniqueName: \"kubernetes.io/projected/b357ce02-37e1-4fc3-bc98-ed990300f797-kube-api-access-d2nqf\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:27.548247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:27.548217 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" Oct 31 01:38:27.548420 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:27.548255 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j" event={"ID":"b357ce02-37e1-4fc3-bc98-ed990300f797","Type":"ContainerDied","Data":"d77d3cf99c7259d0fe3f59d8b5b7420e51afe73a1df9739098940cdf11f9a1ab"} Oct 31 01:38:27.548420 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:27.548302 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d77d3cf99c7259d0fe3f59d8b5b7420e51afe73a1df9739098940cdf11f9a1ab" Oct 31 01:38:32.066690 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:32.066650 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:38:38.549356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.549320 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr"] Oct 31 01:38:38.549903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.549544 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" podUID="6798b3a4-3452-4c64-8fb7-7b26c628193a" containerName="payload-pre-processing" containerID="cri-o://f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d" gracePeriod=30 Oct 31 01:38:38.592017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.591965 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" podUID="6798b3a4-3452-4c64-8fb7-7b26c628193a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.129:9004: connect: connection refused" Oct 31 01:38:38.610960 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:38.610923 5756 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6798b3a4_3452_4c64_8fb7_7b26c628193a.slice/crio-f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d.scope\": RecentStats: unable to find data in memory cache]" Oct 31 01:38:38.831977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.831940 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:38:38.947671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.947630 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume\") pod \"6798b3a4-3452-4c64-8fb7-7b26c628193a\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " Oct 31 01:38:38.947909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.947826 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg\" (UniqueName: \"kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg\") pod \"6798b3a4-3452-4c64-8fb7-7b26c628193a\" (UID: \"6798b3a4-3452-4c64-8fb7-7b26c628193a\") " Oct 31 01:38:38.948165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.948134 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume" pod "6798b3a4-3452-4c64-8fb7-7b26c628193a" (UID: "6798b3a4-3452-4c64-8fb7-7b26c628193a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:38.948291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.948271 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6798b3a4-3452-4c64-8fb7-7b26c628193a-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:38.950296 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:38.950268 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg" pod "6798b3a4-3452-4c64-8fb7-7b26c628193a" (UID: "6798b3a4-3452-4c64-8fb7-7b26c628193a"). InnerVolumeSpecName "kube-api-access-bj7vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:39.050040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.050000 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj7vg\" (UniqueName: \"kubernetes.io/projected/6798b3a4-3452-4c64-8fb7-7b26c628193a-kube-api-access-bj7vg\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:39.622962 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.622925 5756 generic.go:356] "Generic (PLEG): container finished" podID="6798b3a4-3452-4c64-8fb7-7b26c628193a" containerID="f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d" exitCode=0 Oct 31 01:38:39.623459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.622980 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" Oct 31 01:38:39.623459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.622995 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" event={"ID":"6798b3a4-3452-4c64-8fb7-7b26c628193a","Type":"ContainerDied","Data":"f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d"} Oct 31 01:38:39.623459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.623037 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr" event={"ID":"6798b3a4-3452-4c64-8fb7-7b26c628193a","Type":"ContainerDied","Data":"3c08cd604ad7a9b91a74cbb1fb46a1f22e617e3696e5b9acb991e31f33b15f60"} Oct 31 01:38:39.623459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.623058 5756 scope.go:122] "RemoveContainer" containerID="f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d" Oct 31 01:38:39.633111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.633092 5756 scope.go:122] "RemoveContainer" containerID="f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d" Oct 31 01:38:39.633394 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:39.633374 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d\": container with ID starting with f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d not found: ID does not exist" containerID="f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d" Oct 31 01:38:39.633446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.633405 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d"} err="failed to get container status \"f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d\": rpc error: code = NotFound desc = could not find container \"f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d\": container with ID starting with f21ddf9ce7e1ad7645f72c6dba54687ff294d506d377ebd79454e4a6b003631d not found: ID does not exist" Oct 31 01:38:39.784796 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.784760 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr"] Oct 31 01:38:39.834057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:39.834021 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f5b0cb-66d85b9547-vznjr"] Oct 31 01:38:40.216932 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:40.216903 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz"] Oct 31 01:38:40.220873 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:40.220839 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f5b0cb-7g2hz"] Oct 31 01:38:40.311188 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:40.311155 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c0aec2-383d-4831-b8fa-a8a52ff4a4da" path="/var/lib/kubelet/pods/51c0aec2-383d-4831-b8fa-a8a52ff4a4da/volumes" Oct 31 01:38:40.311577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:40.311564 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6798b3a4-3452-4c64-8fb7-7b26c628193a" path="/var/lib/kubelet/pods/6798b3a4-3452-4c64-8fb7-7b26c628193a/volumes" Oct 31 01:38:48.129281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.129251 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht"] Oct 31 01:38:48.130059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.130037 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6798b3a4-3452-4c64-8fb7-7b26c628193a" containerName="payload-pre-processing" Oct 31 01:38:48.130173 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.130059 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6798b3a4-3452-4c64-8fb7-7b26c628193a" containerName="payload-pre-processing" Oct 31 01:38:48.130243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.130181 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b357ce02-37e1-4fc3-bc98-ed990300f797" containerName="revoke-keys" Oct 31 01:38:48.130243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.130192 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b357ce02-37e1-4fc3-bc98-ed990300f797" containerName="revoke-keys" Oct 31 01:38:48.130343 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.130329 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b357ce02-37e1-4fc3-bc98-ed990300f797" containerName="revoke-keys" Oct 31 01:38:48.130393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.130344 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6798b3a4-3452-4c64-8fb7-7b26c628193a" containerName="payload-pre-processing" Oct 31 01:38:48.135223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.135202 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.188758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.188728 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:38:48.238211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.238177 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:38:48.240479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.240457 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht"] Oct 31 01:38:48.256047 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.256014 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltzs\" (UniqueName: \"kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.256165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.256092 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.256254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.256237 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.357635 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.357607 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltzs\" (UniqueName: \"kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.357813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.357667 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.357813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.357751 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.358407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.358383 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.360279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.360254 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.415550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.415483 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltzs\" (UniqueName: \"kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs\") pod \"maas-api-revoke-keys-e2e-ipp-b-634135-822ht\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.449207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.449179 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:48.829927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:48.829905 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht"] Oct 31 01:38:48.831606 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:38:48.831575 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0102c3_c6b3_45e0_9e15_febcfe143165.slice/crio-c228e68900729f24486a207b5d3ceb12466e256b8559f0ac1808fe4f1fbdea8f WatchSource:0}: Error finding container c228e68900729f24486a207b5d3ceb12466e256b8559f0ac1808fe4f1fbdea8f: Status 404 returned error can't find the container with id c228e68900729f24486a207b5d3ceb12466e256b8559f0ac1808fe4f1fbdea8f Oct 31 01:38:49.687473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:49.687440 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" event={"ID":"5c0102c3-c6b3-45e0-9e15-febcfe143165","Type":"ContainerStarted","Data":"8542bbcb244610c133a4467552909c4cdda8c8e9e2359b06e23cc1ed95ce9b47"} Oct 31 01:38:49.687878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:49.687494 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" event={"ID":"5c0102c3-c6b3-45e0-9e15-febcfe143165","Type":"ContainerStarted","Data":"c228e68900729f24486a207b5d3ceb12466e256b8559f0ac1808fe4f1fbdea8f"} Oct 31 01:38:49.805577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:49.805521 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" podStartSLOduration=1.805506675 podStartE2EDuration="1.805506675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:38:49.803362066 +0000 UTC m=+2501.610329610" watchObservedRunningTime="2026-10-31 01:38:49.805506675 +0000 UTC m=+2501.612474218" Oct 31 01:38:50.660569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:50.660537 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:38:50.695851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:50.695814 5756 generic.go:356] "Generic (PLEG): container finished" podID="5c0102c3-c6b3-45e0-9e15-febcfe143165" containerID="8542bbcb244610c133a4467552909c4cdda8c8e9e2359b06e23cc1ed95ce9b47" exitCode=0 Oct 31 01:38:50.696316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:50.695899 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" event={"ID":"5c0102c3-c6b3-45e0-9e15-febcfe143165","Type":"ContainerDied","Data":"8542bbcb244610c133a4467552909c4cdda8c8e9e2359b06e23cc1ed95ce9b47"} Oct 31 01:38:51.842661 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:51.842636 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:52.006465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.006383 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs\" (UniqueName: \"kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs\") pod \"5c0102c3-c6b3-45e0-9e15-febcfe143165\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " Oct 31 01:38:52.006627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.006464 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca\") pod \"5c0102c3-c6b3-45e0-9e15-febcfe143165\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " Oct 31 01:38:52.006627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.006562 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca\") pod \"5c0102c3-c6b3-45e0-9e15-febcfe143165\" (UID: \"5c0102c3-c6b3-45e0-9e15-febcfe143165\") " Oct 31 01:38:52.006968 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.006930 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca" pod "5c0102c3-c6b3-45e0-9e15-febcfe143165" (UID: "5c0102c3-c6b3-45e0-9e15-febcfe143165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:52.008665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.008643 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca" pod "5c0102c3-c6b3-45e0-9e15-febcfe143165" (UID: "5c0102c3-c6b3-45e0-9e15-febcfe143165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:38:52.008749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.008702 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs" pod "5c0102c3-c6b3-45e0-9e15-febcfe143165" (UID: "5c0102c3-c6b3-45e0-9e15-febcfe143165"). InnerVolumeSpecName "kube-api-access-2ltzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:52.108746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.108714 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:52.108746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.108745 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ltzs\" (UniqueName: \"kubernetes.io/projected/5c0102c3-c6b3-45e0-9e15-febcfe143165-kube-api-access-2ltzs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:52.108980 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.108760 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c0102c3-c6b3-45e0-9e15-febcfe143165-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:52.709386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.709312 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" Oct 31 01:38:52.709386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.709329 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht" event={"ID":"5c0102c3-c6b3-45e0-9e15-febcfe143165","Type":"ContainerDied","Data":"c228e68900729f24486a207b5d3ceb12466e256b8559f0ac1808fe4f1fbdea8f"} Oct 31 01:38:52.709386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:52.709373 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c228e68900729f24486a207b5d3ceb12466e256b8559f0ac1808fe4f1fbdea8f" Oct 31 01:38:57.212843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.212806 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm"] Oct 31 01:38:57.213380 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.213072 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" podUID="c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" containerName="payload-processing" containerID="cri-o://0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d" gracePeriod=30 Oct 31 01:38:57.319201 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.319161 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r"] Oct 31 01:38:57.501114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.501085 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:38:57.678842 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.678814 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume\") pod \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " Oct 31 01:38:57.679024 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.678957 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch\" (UniqueName: \"kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch\") pod \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\" (UID: \"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0\") " Oct 31 01:38:57.679254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.679232 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume" pod "c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" (UID: "c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:57.679469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.679454 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:57.681227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.681202 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch" pod "c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" (UID: "c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0"). InnerVolumeSpecName "kube-api-access-7nzch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:57.743431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.743330 5756 generic.go:356] "Generic (PLEG): container finished" podID="c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" containerID="0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d" exitCode=0 Oct 31 01:38:57.743431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.743384 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" Oct 31 01:38:57.743675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.743430 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" event={"ID":"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0","Type":"ContainerDied","Data":"0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d"} Oct 31 01:38:57.743675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.743505 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm" event={"ID":"c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0","Type":"ContainerDied","Data":"ad8e2e0c6e0e4f8fefd1872ea894097814856e5492462110ed7bf5c5bb3367b8"} Oct 31 01:38:57.743675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.743612 5756 scope.go:122] "RemoveContainer" containerID="0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d" Oct 31 01:38:57.743675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.743663 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" podUID="cb5d4b73-f6e4-44ec-9d94-1ad95844a892" containerName="payload-pre-processing" containerID="cri-o://b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd" gracePeriod=30 Oct 31 01:38:57.763467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.763437 5756 scope.go:122] "RemoveContainer" containerID="0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d" Oct 31 01:38:57.763834 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:57.763811 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d\": container with ID starting with 0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d not found: ID does not exist" containerID="0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d" Oct 31 01:38:57.763945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.763842 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d"} err="failed to get container status \"0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d\": rpc error: code = NotFound desc = could not find container \"0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d\": container with ID starting with 0d6df41bec0ec4620eecb2e188622bb303017b277ab34482a584abf3dd93c86d not found: ID does not exist" Oct 31 01:38:57.780201 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:57.780177 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nzch\" (UniqueName: \"kubernetes.io/projected/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0-kube-api-access-7nzch\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.012595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.012560 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:38:58.020739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.020711 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm"] Oct 31 01:38:58.024745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.024718 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7cfab8-69fb96d8dd-62dmm"] Oct 31 01:38:58.084233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.084206 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume\") pod \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " Oct 31 01:38:58.084455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.084297 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w\" (UniqueName: \"kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w\") pod \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\" (UID: \"cb5d4b73-f6e4-44ec-9d94-1ad95844a892\") " Oct 31 01:38:58.084658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.084634 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume" pod "cb5d4b73-f6e4-44ec-9d94-1ad95844a892" (UID: "cb5d4b73-f6e4-44ec-9d94-1ad95844a892"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:58.084775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.084759 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.086508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.086491 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w" pod "cb5d4b73-f6e4-44ec-9d94-1ad95844a892" (UID: "cb5d4b73-f6e4-44ec-9d94-1ad95844a892"). InnerVolumeSpecName "kube-api-access-pjl4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:58.185569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.185535 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjl4w\" (UniqueName: \"kubernetes.io/projected/cb5d4b73-f6e4-44ec-9d94-1ad95844a892-kube-api-access-pjl4w\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.311256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.311226 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" path="/var/lib/kubelet/pods/c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0/volumes" Oct 31 01:38:58.398009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.397976 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479"] Oct 31 01:38:58.398236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.398210 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" podUID="a4e86551-ebed-4a03-8993-5742791926c7" containerName="payload-processing" containerID="cri-o://206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7" gracePeriod=30 Oct 31 01:38:58.448264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.448235 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5"] Oct 31 01:38:58.448504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.448482 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" podUID="5357c559-a6a7-419a-be4e-b26ef903c71a" containerName="payload-pre-processing" containerID="cri-o://5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c" gracePeriod=30 Oct 31 01:38:58.721538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.721515 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:38:58.725099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.725075 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:38:58.752216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.752188 5756 generic.go:356] "Generic (PLEG): container finished" podID="a4e86551-ebed-4a03-8993-5742791926c7" containerID="206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7" exitCode=0 Oct 31 01:38:58.752354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.752234 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" Oct 31 01:38:58.752354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.752280 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" event={"ID":"a4e86551-ebed-4a03-8993-5742791926c7","Type":"ContainerDied","Data":"206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7"} Oct 31 01:38:58.752354 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.752334 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479" event={"ID":"a4e86551-ebed-4a03-8993-5742791926c7","Type":"ContainerDied","Data":"6bc6ed9fb285c31aeb84670cb7f5032916a2c3a2e88e096f8bf34618e4b8473e"} Oct 31 01:38:58.752496 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.752366 5756 scope.go:122] "RemoveContainer" containerID="206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7" Oct 31 01:38:58.753638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.753614 5756 generic.go:356] "Generic (PLEG): container finished" podID="cb5d4b73-f6e4-44ec-9d94-1ad95844a892" containerID="b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd" exitCode=0 Oct 31 01:38:58.753753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.753654 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" Oct 31 01:38:58.753753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.753692 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" event={"ID":"cb5d4b73-f6e4-44ec-9d94-1ad95844a892","Type":"ContainerDied","Data":"b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd"} Oct 31 01:38:58.753753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.753730 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r" event={"ID":"cb5d4b73-f6e4-44ec-9d94-1ad95844a892","Type":"ContainerDied","Data":"6737e391bb00f35bc2f3599369ef0d837402c0200f540c61a502b536eff6c0e1"} Oct 31 01:38:58.755021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.754998 5756 generic.go:356] "Generic (PLEG): container finished" podID="5357c559-a6a7-419a-be4e-b26ef903c71a" containerID="5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c" exitCode=0 Oct 31 01:38:58.755153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.755037 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" Oct 31 01:38:58.755153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.755040 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" event={"ID":"5357c559-a6a7-419a-be4e-b26ef903c71a","Type":"ContainerDied","Data":"5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c"} Oct 31 01:38:58.755250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.755170 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5" event={"ID":"5357c559-a6a7-419a-be4e-b26ef903c71a","Type":"ContainerDied","Data":"9bffeb05aa89662be60425d2943690f6a36998c5aac6a97a7a8f1ed9da108367"} Oct 31 01:38:58.762003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.761986 5756 scope.go:122] "RemoveContainer" containerID="206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7" Oct 31 01:38:58.762272 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:58.762251 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7\": container with ID starting with 206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7 not found: ID does not exist" containerID="206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7" Oct 31 01:38:58.762369 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.762278 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7"} err="failed to get container status \"206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7\": rpc error: code = NotFound desc = could not find container \"206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7\": container with ID starting with 206d797b849b2d8391a94ad8b5b4f9704fc871d4cabdee61b29720c49b815cf7 not found: ID does not exist" Oct 31 01:38:58.762369 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.762293 5756 scope.go:122] "RemoveContainer" containerID="b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd" Oct 31 01:38:58.770701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.770684 5756 scope.go:122] "RemoveContainer" containerID="b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd" Oct 31 01:38:58.770993 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:58.770976 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd\": container with ID starting with b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd not found: ID does not exist" containerID="b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd" Oct 31 01:38:58.771070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.770999 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd"} err="failed to get container status \"b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd\": rpc error: code = NotFound desc = could not find container \"b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd\": container with ID starting with b2cd0de1c2f4890967dfbcdd119f00b0bde9cfdc4922bdb09832c00861bf34fd not found: ID does not exist" Oct 31 01:38:58.771070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.771014 5756 scope.go:122] "RemoveContainer" containerID="5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c" Oct 31 01:38:58.779755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.779737 5756 scope.go:122] "RemoveContainer" containerID="5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c" Oct 31 01:38:58.780104 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:38:58.780081 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c\": container with ID starting with 5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c not found: ID does not exist" containerID="5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c" Oct 31 01:38:58.780165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.780115 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c"} err="failed to get container status \"5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c\": rpc error: code = NotFound desc = could not find container \"5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c\": container with ID starting with 5eb82e4acf7928da67058884c770fca63c594a7600f8f3ac220e72452196fa8c not found: ID does not exist" Oct 31 01:38:58.793585 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.793562 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume\") pod \"a4e86551-ebed-4a03-8993-5742791926c7\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " Oct 31 01:38:58.793679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.793635 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444\" (UniqueName: \"kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444\") pod \"5357c559-a6a7-419a-be4e-b26ef903c71a\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " Oct 31 01:38:58.793712 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.793694 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume\") pod \"5357c559-a6a7-419a-be4e-b26ef903c71a\" (UID: \"5357c559-a6a7-419a-be4e-b26ef903c71a\") " Oct 31 01:38:58.793893 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.793876 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75\" (UniqueName: \"kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75\") pod \"a4e86551-ebed-4a03-8993-5742791926c7\" (UID: \"a4e86551-ebed-4a03-8993-5742791926c7\") " Oct 31 01:38:58.794065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.794044 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume" pod "a4e86551-ebed-4a03-8993-5742791926c7" (UID: "a4e86551-ebed-4a03-8993-5742791926c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:58.794116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.794098 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume" pod "5357c559-a6a7-419a-be4e-b26ef903c71a" (UID: "5357c559-a6a7-419a-be4e-b26ef903c71a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:38:58.794418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.794350 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e86551-ebed-4a03-8993-5742791926c7-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.794418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.794373 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5357c559-a6a7-419a-be4e-b26ef903c71a-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.796067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.796043 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75" pod "a4e86551-ebed-4a03-8993-5742791926c7" (UID: "a4e86551-ebed-4a03-8993-5742791926c7"). InnerVolumeSpecName "kube-api-access-trb75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:58.796148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.796102 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444" pod "5357c559-a6a7-419a-be4e-b26ef903c71a" (UID: "5357c559-a6a7-419a-be4e-b26ef903c71a"). InnerVolumeSpecName "kube-api-access-r8444". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:38:58.895522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.895427 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8444\" (UniqueName: \"kubernetes.io/projected/5357c559-a6a7-419a-be4e-b26ef903c71a-kube-api-access-r8444\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.895522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.895456 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trb75\" (UniqueName: \"kubernetes.io/projected/a4e86551-ebed-4a03-8993-5742791926c7-kube-api-access-trb75\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:38:58.985924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.985884 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc"] Oct 31 01:38:58.990349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:58.990324 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7cfab8-jgglc"] Oct 31 01:38:59.160043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:59.159943 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r"] Oct 31 01:38:59.164132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:59.164109 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7cfab8-7f47847b75-fvj9r"] Oct 31 01:38:59.274961 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:59.274934 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5"] Oct 31 01:38:59.285844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:59.285818 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6c7db9-75cb587d8c-7whk5"] Oct 31 01:38:59.408432 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:59.408399 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479"] Oct 31 01:38:59.417345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:38:59.417283 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c7db9-84d975d5bd-r8479"] Oct 31 01:39:00.311125 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:00.311084 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5357c559-a6a7-419a-be4e-b26ef903c71a" path="/var/lib/kubelet/pods/5357c559-a6a7-419a-be4e-b26ef903c71a/volumes" Oct 31 01:39:00.311528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:00.311514 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f3ad52c-7567-43f4-88d3-514012a4bc70" path="/var/lib/kubelet/pods/6f3ad52c-7567-43f4-88d3-514012a4bc70/volumes" Oct 31 01:39:00.311877 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:00.311850 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e86551-ebed-4a03-8993-5742791926c7" path="/var/lib/kubelet/pods/a4e86551-ebed-4a03-8993-5742791926c7/volumes" Oct 31 01:39:00.312702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:00.312681 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5d4b73-f6e4-44ec-9d94-1ad95844a892" path="/var/lib/kubelet/pods/cb5d4b73-f6e4-44ec-9d94-1ad95844a892/volumes" Oct 31 01:39:00.433888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:00.433833 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j"] Oct 31 01:39:00.483116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:00.483083 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-6c7db9-2c97j"] Oct 31 01:39:02.065654 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:39:02.065621 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:39:02.311827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:02.311792 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b357ce02-37e1-4fc3-bc98-ed990300f797" path="/var/lib/kubelet/pods/b357ce02-37e1-4fc3-bc98-ed990300f797/volumes" Oct 31 01:39:16.922721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.922644 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t"] Oct 31 01:39:16.923139 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923126 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5357c559-a6a7-419a-be4e-b26ef903c71a" containerName="payload-pre-processing" Oct 31 01:39:16.923185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923141 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5357c559-a6a7-419a-be4e-b26ef903c71a" containerName="payload-pre-processing" Oct 31 01:39:16.923185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923167 5756 cpu_manager.go:414] "containerMap: removing container" podUID="a4e86551-ebed-4a03-8993-5742791926c7" containerName="payload-processing" Oct 31 01:39:16.923185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923174 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e86551-ebed-4a03-8993-5742791926c7" containerName="payload-processing" Oct 31 01:39:16.923185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923184 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5c0102c3-c6b3-45e0-9e15-febcfe143165" containerName="revoke-keys" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923189 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c0102c3-c6b3-45e0-9e15-febcfe143165" containerName="revoke-keys" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923203 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cb5d4b73-f6e4-44ec-9d94-1ad95844a892" containerName="payload-pre-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923207 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5d4b73-f6e4-44ec-9d94-1ad95844a892" containerName="payload-pre-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923225 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" containerName="payload-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923230 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" containerName="payload-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923300 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5d4b73-f6e4-44ec-9d94-1ad95844a892" containerName="payload-pre-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923311 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ac98f5-3a81-4fae-8f27-ed2b043fbdf0" containerName="payload-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923320 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5357c559-a6a7-419a-be4e-b26ef903c71a" containerName="payload-pre-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923328 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e86551-ebed-4a03-8993-5742791926c7" containerName="payload-processing" Oct 31 01:39:16.923345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.923335 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c0102c3-c6b3-45e0-9e15-febcfe143165" containerName="revoke-keys" Oct 31 01:39:16.928241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.928222 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:16.981458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.981431 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:39:16.984043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:16.984021 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:39:17.011758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.011717 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bkn\" (UniqueName: \"kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.012000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.011975 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.012095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.012040 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.033107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.033084 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t"] Oct 31 01:39:17.113329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.113290 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.113502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.113345 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.113502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.113379 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bkn\" (UniqueName: \"kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.114572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.114541 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.116625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.116601 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.170811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.170783 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bkn\" (UniqueName: \"kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.241240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.241160 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:17.617523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.617498 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t"] Oct 31 01:39:17.619413 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:39:17.619380 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fabf9d_20cb_4c02_bc35_9449b434f9c0.slice/crio-7c1a46feb808f4ebca0a1133382c960096fa193b388abbd3cf4170155bca2466 WatchSource:0}: Error finding container 7c1a46feb808f4ebca0a1133382c960096fa193b388abbd3cf4170155bca2466: Status 404 returned error can't find the container with id 7c1a46feb808f4ebca0a1133382c960096fa193b388abbd3cf4170155bca2466 Oct 31 01:39:17.883973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:17.883892 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" event={"ID":"b8fabf9d-20cb-4c02-bc35-9449b434f9c0","Type":"ContainerStarted","Data":"7c1a46feb808f4ebca0a1133382c960096fa193b388abbd3cf4170155bca2466"} Oct 31 01:39:18.891177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:18.891145 5756 generic.go:356] "Generic (PLEG): container finished" podID="b8fabf9d-20cb-4c02-bc35-9449b434f9c0" containerID="50c4bd97a0ccdc026d8b976c243c8264d6a868d0e107ea8c899bf9dc113444e1" exitCode=0 Oct 31 01:39:18.891572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:18.891218 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" event={"ID":"b8fabf9d-20cb-4c02-bc35-9449b434f9c0","Type":"ContainerDied","Data":"50c4bd97a0ccdc026d8b976c243c8264d6a868d0e107ea8c899bf9dc113444e1"} Oct 31 01:39:20.039169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.039139 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:20.145317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.145282 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca\") pod \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " Oct 31 01:39:20.145490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.145344 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca\") pod \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " Oct 31 01:39:20.145490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.145432 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn\" (UniqueName: \"kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn\") pod \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\" (UID: \"b8fabf9d-20cb-4c02-bc35-9449b434f9c0\") " Oct 31 01:39:20.145691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.145669 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca" pod "b8fabf9d-20cb-4c02-bc35-9449b434f9c0" (UID: "b8fabf9d-20cb-4c02-bc35-9449b434f9c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:39:20.145852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.145838 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:20.147658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.147607 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca" pod "b8fabf9d-20cb-4c02-bc35-9449b434f9c0" (UID: "b8fabf9d-20cb-4c02-bc35-9449b434f9c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:39:20.147735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.147667 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn" pod "b8fabf9d-20cb-4c02-bc35-9449b434f9c0" (UID: "b8fabf9d-20cb-4c02-bc35-9449b434f9c0"). InnerVolumeSpecName "kube-api-access-m9bkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:39:20.246944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.246909 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:20.246944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.246937 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bkn\" (UniqueName: \"kubernetes.io/projected/b8fabf9d-20cb-4c02-bc35-9449b434f9c0-kube-api-access-m9bkn\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:20.903469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.903434 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" event={"ID":"b8fabf9d-20cb-4c02-bc35-9449b434f9c0","Type":"ContainerDied","Data":"7c1a46feb808f4ebca0a1133382c960096fa193b388abbd3cf4170155bca2466"} Oct 31 01:39:20.903469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.903454 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t" Oct 31 01:39:20.903714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:20.903481 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c1a46feb808f4ebca0a1133382c960096fa193b388abbd3cf4170155bca2466" Oct 31 01:39:23.083931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:23.083900 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:39:25.265529 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.265500 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc"] Oct 31 01:39:25.266011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.265998 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b8fabf9d-20cb-4c02-bc35-9449b434f9c0" containerName="revoke-keys" Oct 31 01:39:25.266063 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.266012 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8fabf9d-20cb-4c02-bc35-9449b434f9c0" containerName="revoke-keys" Oct 31 01:39:25.266107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.266097 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8fabf9d-20cb-4c02-bc35-9449b434f9c0" containerName="revoke-keys" Oct 31 01:39:25.270280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.270265 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.302990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.302961 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.303136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.302994 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.303136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.303104 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj9z\" (UniqueName: \"kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.375132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.375103 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:39:25.375296 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.375188 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:39:25.377383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.377361 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc"] Oct 31 01:39:25.404002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.403971 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.404002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.404012 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.404259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.404053 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj9z\" (UniqueName: \"kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.406040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.405226 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.413466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.407793 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.467425 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.467391 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj9z\" (UniqueName: \"kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z\") pod \"maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.583625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.583593 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:25.959701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:25.959674 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc"] Oct 31 01:39:25.961981 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:39:25.961954 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5353d7_b3bd_49f3_99af_9971e8795495.slice/crio-486af561ec43e6fa3aefce0977bd691df2dd35b7ba4419bacc852373b933932c WatchSource:0}: Error finding container 486af561ec43e6fa3aefce0977bd691df2dd35b7ba4419bacc852373b933932c: Status 404 returned error can't find the container with id 486af561ec43e6fa3aefce0977bd691df2dd35b7ba4419bacc852373b933932c Oct 31 01:39:26.939704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:26.939664 5756 generic.go:356] "Generic (PLEG): container finished" podID="6e5353d7-b3bd-49f3-99af-9971e8795495" containerID="9c79d8ef18dfbb46c0d70e57d2036e7980321762a917c94e71ebf64db385d6ab" exitCode=0 Oct 31 01:39:26.940127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:26.939715 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" event={"ID":"6e5353d7-b3bd-49f3-99af-9971e8795495","Type":"ContainerDied","Data":"9c79d8ef18dfbb46c0d70e57d2036e7980321762a917c94e71ebf64db385d6ab"} Oct 31 01:39:26.940127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:26.939755 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" event={"ID":"6e5353d7-b3bd-49f3-99af-9971e8795495","Type":"ContainerStarted","Data":"486af561ec43e6fa3aefce0977bd691df2dd35b7ba4419bacc852373b933932c"} Oct 31 01:39:28.098286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.098261 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:28.134169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.134143 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca\") pod \"6e5353d7-b3bd-49f3-99af-9971e8795495\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " Oct 31 01:39:28.134320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.134190 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca\") pod \"6e5353d7-b3bd-49f3-99af-9971e8795495\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " Oct 31 01:39:28.134386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.134353 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z\" (UniqueName: \"kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z\") pod \"6e5353d7-b3bd-49f3-99af-9971e8795495\" (UID: \"6e5353d7-b3bd-49f3-99af-9971e8795495\") " Oct 31 01:39:28.134583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.134558 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca" pod "6e5353d7-b3bd-49f3-99af-9971e8795495" (UID: "6e5353d7-b3bd-49f3-99af-9971e8795495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:39:28.134658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.134649 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:28.136405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.136383 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca" pod "6e5353d7-b3bd-49f3-99af-9971e8795495" (UID: "6e5353d7-b3bd-49f3-99af-9971e8795495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:39:28.136457 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.136400 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z" pod "6e5353d7-b3bd-49f3-99af-9971e8795495" (UID: "6e5353d7-b3bd-49f3-99af-9971e8795495"). InnerVolumeSpecName "kube-api-access-6qj9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:39:28.236146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.236067 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qj9z\" (UniqueName: \"kubernetes.io/projected/6e5353d7-b3bd-49f3-99af-9971e8795495-kube-api-access-6qj9z\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:28.236146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.236092 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e5353d7-b3bd-49f3-99af-9971e8795495-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:28.952362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.952332 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" Oct 31 01:39:28.952550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.952377 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc" event={"ID":"6e5353d7-b3bd-49f3-99af-9971e8795495","Type":"ContainerDied","Data":"486af561ec43e6fa3aefce0977bd691df2dd35b7ba4419bacc852373b933932c"} Oct 31 01:39:28.952550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:28.952424 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="486af561ec43e6fa3aefce0977bd691df2dd35b7ba4419bacc852373b933932c" Oct 31 01:39:32.066192 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:39:32.066155 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:39:33.687021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.686986 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9"] Oct 31 01:39:33.687717 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.687694 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6e5353d7-b3bd-49f3-99af-9971e8795495" containerName="revoke-keys" Oct 31 01:39:33.687814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.687719 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e5353d7-b3bd-49f3-99af-9971e8795495" containerName="revoke-keys" Oct 31 01:39:33.687912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.687881 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e5353d7-b3bd-49f3-99af-9971e8795495" containerName="revoke-keys" Oct 31 01:39:33.691104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.691082 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:33.785636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.785597 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3f0818\"" Oct 31 01:39:33.804567 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.804529 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9"] Oct 31 01:39:33.804725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.804668 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3f0818-dockercfg-rh8n8\"" Oct 31 01:39:33.805335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.805308 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:33.805508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.805491 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jks9m\" (UniqueName: \"kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m\") pod \"payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:33.906250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.906210 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jks9m\" (UniqueName: \"kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m\") pod \"payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:33.906454 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.906340 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:33.907349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.907324 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:33.977079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:33.976624 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jks9m\" (UniqueName: \"kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m\") pod \"payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:34.008150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:34.008108 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:34.388342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:34.388317 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9"] Oct 31 01:39:34.390265 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:39:34.390235 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c53b9b_55bb_4661_86d2_5255f63309e9.slice/crio-ca041f6a1fd31aef94c3a0d8e587bb79d9cd7e3116bd4810bba0a898c69b99e5 WatchSource:0}: Error finding container ca041f6a1fd31aef94c3a0d8e587bb79d9cd7e3116bd4810bba0a898c69b99e5: Status 404 returned error can't find the container with id ca041f6a1fd31aef94c3a0d8e587bb79d9cd7e3116bd4810bba0a898c69b99e5 Oct 31 01:39:34.991391 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:34.991350 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" event={"ID":"e6c53b9b-55bb-4661-86d2-5255f63309e9","Type":"ContainerStarted","Data":"ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b"} Oct 31 01:39:34.991852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:34.991410 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" event={"ID":"e6c53b9b-55bb-4661-86d2-5255f63309e9","Type":"ContainerStarted","Data":"ca041f6a1fd31aef94c3a0d8e587bb79d9cd7e3116bd4810bba0a898c69b99e5"} Oct 31 01:39:34.991852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:34.991436 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:35.113026 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:35.112964 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" podStartSLOduration=2.112946787 podStartE2EDuration="2.112946787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:35.111181268 +0000 UTC m=+2546.918148811" watchObservedRunningTime="2026-10-31 01:39:35.112946787 +0000 UTC m=+2546.919914331" Oct 31 01:39:35.865479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:35.865444 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t"] Oct 31 01:39:35.910699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:35.910668 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ea963-ck88t"] Oct 31 01:39:36.310972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:36.310944 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8fabf9d-20cb-4c02-bc35-9449b434f9c0" path="/var/lib/kubelet/pods/b8fabf9d-20cb-4c02-bc35-9449b434f9c0/volumes" Oct 31 01:39:45.999648 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:45.999615 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:39:51.619979 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:51.619945 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:39:53.401060 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:53.401013 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll"] Oct 31 01:39:53.401487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:53.401278 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" containerID="cri-o://a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684" gracePeriod=30 Oct 31 01:39:58.669146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.669084 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:39:58.794466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794434 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794490 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794532 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794563 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794593 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794639 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794678 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794772 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.794927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794819 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data\") pod \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\" (UID: \"949b38b8-99b4-4da4-96d8-9ff1bd5d0557\") " Oct 31 01:39:58.795106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794927 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:39:58.795106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.794944 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:39:58.795227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795133 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:39:58.795227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795148 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:39:58.795345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795305 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.795345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795323 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.795345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795332 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.795666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795344 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.795666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.795496 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:39:58.796967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.796940 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:39:58.797074 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.796963 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "kube-api-access-q6tvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:39:58.797074 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.797020 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:39:58.797250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.797232 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo" pod "949b38b8-99b4-4da4-96d8-9ff1bd5d0557" (UID: "949b38b8-99b4-4da4-96d8-9ff1bd5d0557"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:39:58.896388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.896357 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.896388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.896383 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.896388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.896393 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.896613 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.896402 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:58.896613 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:58.896412 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6tvx\" (UniqueName: \"kubernetes.io/projected/949b38b8-99b4-4da4-96d8-9ff1bd5d0557-kube-api-access-q6tvx\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:39:59.132622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.132592 5756 generic.go:356] "Generic (PLEG): container finished" podID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerID="a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684" exitCode=0 Oct 31 01:39:59.132803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.132661 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" Oct 31 01:39:59.132803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.132683 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" event={"ID":"949b38b8-99b4-4da4-96d8-9ff1bd5d0557","Type":"ContainerDied","Data":"a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684"} Oct 31 01:39:59.132803 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.132746 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll" event={"ID":"949b38b8-99b4-4da4-96d8-9ff1bd5d0557","Type":"ContainerDied","Data":"6af8cde1159b8269840da7c35e352f94c7d14bd33259c8880048344f14775b83"} Oct 31 01:39:59.132946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.132843 5756 scope.go:122] "RemoveContainer" containerID="a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684" Oct 31 01:39:59.143390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.143369 5756 scope.go:122] "RemoveContainer" containerID="b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a" Oct 31 01:39:59.156217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.156200 5756 scope.go:122] "RemoveContainer" containerID="a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684" Oct 31 01:39:59.156505 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:39:59.156486 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684\": container with ID starting with a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684 not found: ID does not exist" containerID="a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684" Oct 31 01:39:59.156600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.156520 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684"} err="failed to get container status \"a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684\": rpc error: code = NotFound desc = could not find container \"a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684\": container with ID starting with a355d4d502181d1360dad4e58e6b446dbf3d9b61313b743e5478982b1591e684 not found: ID does not exist" Oct 31 01:39:59.156600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.156546 5756 scope.go:122] "RemoveContainer" containerID="b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a" Oct 31 01:39:59.156840 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:39:59.156808 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a\": container with ID starting with b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a not found: ID does not exist" containerID="b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a" Oct 31 01:39:59.156971 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.156848 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a"} err="failed to get container status \"b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a\": rpc error: code = NotFound desc = could not find container \"b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a\": container with ID starting with b7ebb1990752d7f6486c736d15bafba58f4834994468c183055c7624d4dad35a not found: ID does not exist" Oct 31 01:39:59.306002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.305970 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll"] Oct 31 01:39:59.354836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:39:59.354808 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-0ea963-openshift-default-f86f4c75c-rjhll"] Oct 31 01:40:00.310728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:00.310695 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" path="/var/lib/kubelet/pods/949b38b8-99b4-4da4-96d8-9ff1bd5d0557/volumes" Oct 31 01:40:02.066765 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:40:02.066724 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:40:03.666596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:03.666561 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4"] Oct 31 01:40:03.667077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:03.666767 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" podUID="99a3ee11-a307-49b7-b46f-a4ada82148bd" containerName="payload-processing" containerID="cri-o://eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db" gracePeriod=30 Oct 31 01:40:03.951973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:03.951946 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:40:04.054088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.054055 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq\" (UniqueName: \"kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq\") pod \"99a3ee11-a307-49b7-b46f-a4ada82148bd\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " Oct 31 01:40:04.054263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.054171 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume\") pod \"99a3ee11-a307-49b7-b46f-a4ada82148bd\" (UID: \"99a3ee11-a307-49b7-b46f-a4ada82148bd\") " Oct 31 01:40:04.054522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.054487 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume" pod "99a3ee11-a307-49b7-b46f-a4ada82148bd" (UID: "99a3ee11-a307-49b7-b46f-a4ada82148bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:04.056347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.056326 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq" pod "99a3ee11-a307-49b7-b46f-a4ada82148bd" (UID: "99a3ee11-a307-49b7-b46f-a4ada82148bd"). InnerVolumeSpecName "kube-api-access-gxkpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:04.155334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.155298 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxkpq\" (UniqueName: \"kubernetes.io/projected/99a3ee11-a307-49b7-b46f-a4ada82148bd-kube-api-access-gxkpq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:04.155334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.155330 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99a3ee11-a307-49b7-b46f-a4ada82148bd-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:04.163502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.163465 5756 generic.go:356] "Generic (PLEG): container finished" podID="99a3ee11-a307-49b7-b46f-a4ada82148bd" containerID="eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db" exitCode=0 Oct 31 01:40:04.163697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.163533 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" Oct 31 01:40:04.163697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.163591 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" event={"ID":"99a3ee11-a307-49b7-b46f-a4ada82148bd","Type":"ContainerDied","Data":"eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db"} Oct 31 01:40:04.163697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.163667 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4" event={"ID":"99a3ee11-a307-49b7-b46f-a4ada82148bd","Type":"ContainerDied","Data":"31c0e84524ab4523f0338502b90bc709a93abe638f5a9c7c32bc4709b8fbea72"} Oct 31 01:40:04.163822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.163712 5756 scope.go:122] "RemoveContainer" containerID="eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db" Oct 31 01:40:04.177810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.177789 5756 scope.go:122] "RemoveContainer" containerID="eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db" Oct 31 01:40:04.178302 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:40:04.178260 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db\": container with ID starting with eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db not found: ID does not exist" containerID="eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db" Oct 31 01:40:04.178428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.178309 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db"} err="failed to get container status \"eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db\": rpc error: code = NotFound desc = could not find container \"eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db\": container with ID starting with eb9e913086ed163528d7f97c4480d275cbf563474e898b7cdded89fb3b6808db not found: ID does not exist" Oct 31 01:40:04.326108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.326078 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4"] Oct 31 01:40:04.329536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:04.329508 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-634135-56686b8fcb-crwc4"] Oct 31 01:40:05.371055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:05.371017 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht"] Oct 31 01:40:05.420210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:05.420171 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-634135-822ht"] Oct 31 01:40:06.312489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.312460 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c0102c3-c6b3-45e0-9e15-febcfe143165" path="/var/lib/kubelet/pods/5c0102c3-c6b3-45e0-9e15-febcfe143165/volumes" Oct 31 01:40:06.312832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.312818 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a3ee11-a307-49b7-b46f-a4ada82148bd" path="/var/lib/kubelet/pods/99a3ee11-a307-49b7-b46f-a4ada82148bd/volumes" Oct 31 01:40:06.916177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916141 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4"] Oct 31 01:40:06.916625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916611 5756 cpu_manager.go:414] "containerMap: removing container" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" Oct 31 01:40:06.916683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916627 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" Oct 31 01:40:06.916683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916641 5756 cpu_manager.go:414] "containerMap: removing container" podUID="99a3ee11-a307-49b7-b46f-a4ada82148bd" containerName="payload-processing" Oct 31 01:40:06.916683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916646 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a3ee11-a307-49b7-b46f-a4ada82148bd" containerName="payload-processing" Oct 31 01:40:06.916792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916733 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a3ee11-a307-49b7-b46f-a4ada82148bd" containerName="payload-processing" Oct 31 01:40:06.916792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916749 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" Oct 31 01:40:06.916792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.916760 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" Oct 31 01:40:06.921095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.921080 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:06.978706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:06.978661 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4"] Oct 31 01:40:07.025252 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.025221 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:40:07.025252 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.025228 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:40:07.088144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.088100 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.088333 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.088232 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsrg\" (UniqueName: \"kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.088396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.088341 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.189416 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.189330 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.189416 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.189386 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsrg\" (UniqueName: \"kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.189664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.189448 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.190190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.190167 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.191839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.191818 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.246531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.246500 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsrg\" (UniqueName: \"kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.533313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.533287 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:07.907191 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:07.907068 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4"] Oct 31 01:40:07.910101 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:07.910063 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1674753c_59af_4074_b3f5_355c772fb44d.slice/crio-17362a8205d9c5a805a55d9303bd4d71b4649d9bf3ebbb3908a72cbdbf7d77ac WatchSource:0}: Error finding container 17362a8205d9c5a805a55d9303bd4d71b4649d9bf3ebbb3908a72cbdbf7d77ac: Status 404 returned error can't find the container with id 17362a8205d9c5a805a55d9303bd4d71b4649d9bf3ebbb3908a72cbdbf7d77ac Oct 31 01:40:08.189850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:08.189769 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" event={"ID":"1674753c-59af-4074-b3f5-355c772fb44d","Type":"ContainerStarted","Data":"17362a8205d9c5a805a55d9303bd4d71b4649d9bf3ebbb3908a72cbdbf7d77ac"} Oct 31 01:40:09.197555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:09.197463 5756 generic.go:356] "Generic (PLEG): container finished" podID="1674753c-59af-4074-b3f5-355c772fb44d" containerID="a4d5658d613f847c2c7fa61db169597007d365f2c35f5ca2756e8942520e3e1a" exitCode=0 Oct 31 01:40:09.197555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:09.197548 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" event={"ID":"1674753c-59af-4074-b3f5-355c772fb44d","Type":"ContainerDied","Data":"a4d5658d613f847c2c7fa61db169597007d365f2c35f5ca2756e8942520e3e1a"} Oct 31 01:40:10.352590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.352564 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:10.526408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.526327 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca\") pod \"1674753c-59af-4074-b3f5-355c772fb44d\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " Oct 31 01:40:10.526589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.526425 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca\") pod \"1674753c-59af-4074-b3f5-355c772fb44d\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " Oct 31 01:40:10.526589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.526495 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg\" (UniqueName: \"kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg\") pod \"1674753c-59af-4074-b3f5-355c772fb44d\" (UID: \"1674753c-59af-4074-b3f5-355c772fb44d\") " Oct 31 01:40:10.526796 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.526773 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca" pod "1674753c-59af-4074-b3f5-355c772fb44d" (UID: "1674753c-59af-4074-b3f5-355c772fb44d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:10.528658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.528626 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca" pod "1674753c-59af-4074-b3f5-355c772fb44d" (UID: "1674753c-59af-4074-b3f5-355c772fb44d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:10.528767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.528701 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg" pod "1674753c-59af-4074-b3f5-355c772fb44d" (UID: "1674753c-59af-4074-b3f5-355c772fb44d"). InnerVolumeSpecName "kube-api-access-rrsrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:10.628285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.628251 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrsrg\" (UniqueName: \"kubernetes.io/projected/1674753c-59af-4074-b3f5-355c772fb44d-kube-api-access-rrsrg\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:10.628285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.628281 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:10.628285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:10.628290 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1674753c-59af-4074-b3f5-355c772fb44d-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:11.210884 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:11.210842 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" Oct 31 01:40:11.211043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:11.210899 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4" event={"ID":"1674753c-59af-4074-b3f5-355c772fb44d","Type":"ContainerDied","Data":"17362a8205d9c5a805a55d9303bd4d71b4649d9bf3ebbb3908a72cbdbf7d77ac"} Oct 31 01:40:11.211043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:11.210953 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17362a8205d9c5a805a55d9303bd4d71b4649d9bf3ebbb3908a72cbdbf7d77ac" Oct 31 01:40:22.928558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.928522 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf"] Oct 31 01:40:22.929736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.929709 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1674753c-59af-4074-b3f5-355c772fb44d" containerName="revoke-keys" Oct 31 01:40:22.929736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.929734 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1674753c-59af-4074-b3f5-355c772fb44d" containerName="revoke-keys" Oct 31 01:40:22.929975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.929761 5756 cpu_manager.go:414] "containerMap: removing container" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" Oct 31 01:40:22.929975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.929766 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949b38b8-99b4-4da4-96d8-9ff1bd5d0557" containerName="istio-proxy" Oct 31 01:40:22.929975 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.929903 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1674753c-59af-4074-b3f5-355c772fb44d" containerName="revoke-keys" Oct 31 01:40:22.933669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.933649 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:22.987164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.987127 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-25b9ff\"" Oct 31 01:40:22.998165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.998129 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42tc\" (UniqueName: \"kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc\") pod \"payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:22.998362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:22.998301 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:23.057312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.057270 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-25b9ff-dockercfg-nx9dc\"" Oct 31 01:40:23.066811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.066779 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf"] Oct 31 01:40:23.099701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.099666 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42tc\" (UniqueName: \"kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc\") pod \"payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:23.099944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.099823 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:23.100646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.100623 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:23.180827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.180750 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42tc\" (UniqueName: \"kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc\") pod \"payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:23.250256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.250225 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:23.642074 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:23.642042 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf"] Oct 31 01:40:23.644305 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:23.644274 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51432c69_09be_4042_9f50_dae704e0cd2c.slice/crio-2607f7814f5ae9efbbd011cfdf58c8b926fc26aaa364096d30d4a47d0dc311c9 WatchSource:0}: Error finding container 2607f7814f5ae9efbbd011cfdf58c8b926fc26aaa364096d30d4a47d0dc311c9: Status 404 returned error can't find the container with id 2607f7814f5ae9efbbd011cfdf58c8b926fc26aaa364096d30d4a47d0dc311c9 Oct 31 01:40:24.296773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:24.296712 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" event={"ID":"51432c69-09be-4042-9f50-dae704e0cd2c","Type":"ContainerStarted","Data":"51bfd5b3092bf50d205f7cba9e08e3da4c9b890afc9536fcd35b7492b0f12845"} Oct 31 01:40:24.297221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:24.296787 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" event={"ID":"51432c69-09be-4042-9f50-dae704e0cd2c","Type":"ContainerStarted","Data":"2607f7814f5ae9efbbd011cfdf58c8b926fc26aaa364096d30d4a47d0dc311c9"} Oct 31 01:40:24.297221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:24.296900 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:24.418182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:24.418128 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" podStartSLOduration=2.418112836 podStartE2EDuration="2.418112836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:24.414225549 +0000 UTC m=+2596.221193093" watchObservedRunningTime="2026-10-31 01:40:24.418112836 +0000 UTC m=+2596.225080378" Oct 31 01:40:25.643313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:25.643273 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9"] Oct 31 01:40:25.643708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:25.643495 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" podUID="e6c53b9b-55bb-4661-86d2-5255f63309e9" containerName="payload-processing" containerID="cri-o://ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b" gracePeriod=30 Oct 31 01:40:25.920799 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:25.920769 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:40:26.032890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.032849 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume\") pod \"e6c53b9b-55bb-4661-86d2-5255f63309e9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " Oct 31 01:40:26.033082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.032990 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m\" (UniqueName: \"kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m\") pod \"e6c53b9b-55bb-4661-86d2-5255f63309e9\" (UID: \"e6c53b9b-55bb-4661-86d2-5255f63309e9\") " Oct 31 01:40:26.033286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.033264 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume" pod "e6c53b9b-55bb-4661-86d2-5255f63309e9" (UID: "e6c53b9b-55bb-4661-86d2-5255f63309e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:26.033506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.033489 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c53b9b-55bb-4661-86d2-5255f63309e9-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:26.035287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.035264 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m" pod "e6c53b9b-55bb-4661-86d2-5255f63309e9" (UID: "e6c53b9b-55bb-4661-86d2-5255f63309e9"). InnerVolumeSpecName "kube-api-access-jks9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:26.135092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.135065 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jks9m\" (UniqueName: \"kubernetes.io/projected/e6c53b9b-55bb-4661-86d2-5255f63309e9-kube-api-access-jks9m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:26.312132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.312095 5756 generic.go:356] "Generic (PLEG): container finished" podID="e6c53b9b-55bb-4661-86d2-5255f63309e9" containerID="ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b" exitCode=0 Oct 31 01:40:26.312336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.312139 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" Oct 31 01:40:26.312336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.312184 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" event={"ID":"e6c53b9b-55bb-4661-86d2-5255f63309e9","Type":"ContainerDied","Data":"ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b"} Oct 31 01:40:26.312336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.312229 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9" event={"ID":"e6c53b9b-55bb-4661-86d2-5255f63309e9","Type":"ContainerDied","Data":"ca041f6a1fd31aef94c3a0d8e587bb79d9cd7e3116bd4810bba0a898c69b99e5"} Oct 31 01:40:26.312336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.312253 5756 scope.go:122] "RemoveContainer" containerID="ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b" Oct 31 01:40:26.323319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.323296 5756 scope.go:122] "RemoveContainer" containerID="ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b" Oct 31 01:40:26.323606 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:40:26.323584 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b\": container with ID starting with ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b not found: ID does not exist" containerID="ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b" Oct 31 01:40:26.323665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.323615 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b"} err="failed to get container status \"ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b\": rpc error: code = NotFound desc = could not find container \"ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b\": container with ID starting with ec776a806d6e29b661238b2af6c2d918cde1bcd6de51f41df0bf86f09742b05b not found: ID does not exist" Oct 31 01:40:26.479908 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.479853 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9"] Oct 31 01:40:26.488173 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:26.488148 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3f0818-5b7cbd75fb-28gh9"] Oct 31 01:40:27.364215 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:27.364161 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:40:27.559599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:27.559559 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc"] Oct 31 01:40:27.563218 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:27.563190 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3f0818-sszlc"] Oct 31 01:40:28.312916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:28.312842 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e5353d7-b3bd-49f3-99af-9971e8795495" path="/var/lib/kubelet/pods/6e5353d7-b3bd-49f3-99af-9971e8795495/volumes" Oct 31 01:40:28.313469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:28.313449 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c53b9b-55bb-4661-86d2-5255f63309e9" path="/var/lib/kubelet/pods/e6c53b9b-55bb-4661-86d2-5255f63309e9/volumes" Oct 31 01:40:31.772714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.772678 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w"] Oct 31 01:40:31.773365 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.773344 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e6c53b9b-55bb-4661-86d2-5255f63309e9" containerName="payload-processing" Oct 31 01:40:31.773452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.773368 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c53b9b-55bb-4661-86d2-5255f63309e9" containerName="payload-processing" Oct 31 01:40:31.773547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.773534 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c53b9b-55bb-4661-86d2-5255f63309e9" containerName="payload-processing" Oct 31 01:40:31.776695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.776674 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:31.891153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.891117 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d867bb-dockercfg-9vsbj\"" Oct 31 01:40:31.891333 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.891164 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d867bb\"" Oct 31 01:40:31.893226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.893203 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w"] Oct 31 01:40:31.900323 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.900297 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:31.900469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:31.900430 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qn72\" (UniqueName: \"kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72\") pod \"payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:32.011147 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.011049 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:32.011956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.011934 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:32.014972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.014942 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qn72\" (UniqueName: \"kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72\") pod \"payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:32.034628 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.034593 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l"] Oct 31 01:40:32.039349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.039327 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.068164 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:40:32.068124 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:40:32.073106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.073077 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qn72\" (UniqueName: \"kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72\") pod \"payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:32.089114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.089087 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:32.094770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.094298 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:40:32.094770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.094445 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:40:32.101771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.101745 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l"] Oct 31 01:40:32.217060 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.217029 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.217228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.217081 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfmk\" (UniqueName: \"kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.217228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.217217 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.318843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.318808 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.319082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.318976 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.319082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.319038 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfmk\" (UniqueName: \"kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.319458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.319440 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.321477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.321458 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.381008 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.380974 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfmk\" (UniqueName: \"kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk\") pod \"maas-api-revoke-sub-f758a2a6-th42l\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:32.502518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.502477 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w"] Oct 31 01:40:32.504419 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:32.504392 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ac59c2_43f5_484f_914c_c9edcbfaf0ad.slice/crio-37e2b22404a3514c94ef0c51c082ac0069cb3d8b23c9f9e49ebd8d6089c1e929 WatchSource:0}: Error finding container 37e2b22404a3514c94ef0c51c082ac0069cb3d8b23c9f9e49ebd8d6089c1e929: Status 404 returned error can't find the container with id 37e2b22404a3514c94ef0c51c082ac0069cb3d8b23c9f9e49ebd8d6089c1e929 Oct 31 01:40:32.654634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:32.654559 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:33.047903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.038006 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l"] Oct 31 01:40:33.047903 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:33.045421 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dc0018_4a87_42ce_850d_91e328b9d54b.slice/crio-cf184d627e3a1a4e73e8a49585b473818fccfbb87a7f81584ea77dd843bb6b74 WatchSource:0}: Error finding container cf184d627e3a1a4e73e8a49585b473818fccfbb87a7f81584ea77dd843bb6b74: Status 404 returned error can't find the container with id cf184d627e3a1a4e73e8a49585b473818fccfbb87a7f81584ea77dd843bb6b74 Oct 31 01:40:33.355843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.355812 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" event={"ID":"35dc0018-4a87-42ce-850d-91e328b9d54b","Type":"ContainerStarted","Data":"57f4a9089a36b6bb3a8393151b595ea7fac870cdf6595f2d0b20c4674a6cd81e"} Oct 31 01:40:33.355993 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.355883 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" event={"ID":"35dc0018-4a87-42ce-850d-91e328b9d54b","Type":"ContainerStarted","Data":"cf184d627e3a1a4e73e8a49585b473818fccfbb87a7f81584ea77dd843bb6b74"} Oct 31 01:40:33.357585 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.357556 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" event={"ID":"76ac59c2-43f5-484f-914c-c9edcbfaf0ad","Type":"ContainerStarted","Data":"269c2ed0ce0da14c9958a76a7c3af3b8fe2cc4d0a66f73f8e1975510ec899796"} Oct 31 01:40:33.357705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.357600 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" event={"ID":"76ac59c2-43f5-484f-914c-c9edcbfaf0ad","Type":"ContainerStarted","Data":"37e2b22404a3514c94ef0c51c082ac0069cb3d8b23c9f9e49ebd8d6089c1e929"} Oct 31 01:40:33.357705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.357651 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:33.471196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.471141 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" podStartSLOduration=2.471126817 podStartE2EDuration="2.471126817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:33.469378604 +0000 UTC m=+2605.276346148" watchObservedRunningTime="2026-10-31 01:40:33.471126817 +0000 UTC m=+2605.278094359" Oct 31 01:40:33.585309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:33.585237 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" podStartSLOduration=2.585217956 podStartE2EDuration="2.585217956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:33.582973806 +0000 UTC m=+2605.389941349" watchObservedRunningTime="2026-10-31 01:40:33.585217956 +0000 UTC m=+2605.392185502" Oct 31 01:40:34.365251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:34.365216 5756 generic.go:356] "Generic (PLEG): container finished" podID="35dc0018-4a87-42ce-850d-91e328b9d54b" containerID="57f4a9089a36b6bb3a8393151b595ea7fac870cdf6595f2d0b20c4674a6cd81e" exitCode=0 Oct 31 01:40:34.365666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:34.365310 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" event={"ID":"35dc0018-4a87-42ce-850d-91e328b9d54b","Type":"ContainerDied","Data":"57f4a9089a36b6bb3a8393151b595ea7fac870cdf6595f2d0b20c4674a6cd81e"} Oct 31 01:40:34.711025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:34.710942 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp"] Oct 31 01:40:34.711232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:34.711196 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" containerID="cri-o://48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d" gracePeriod=30 Oct 31 01:40:35.309326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.309299 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:40:35.514487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.514457 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:35.557514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.557479 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca\") pod \"35dc0018-4a87-42ce-850d-91e328b9d54b\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " Oct 31 01:40:35.557716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.557541 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk\" (UniqueName: \"kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk\") pod \"35dc0018-4a87-42ce-850d-91e328b9d54b\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " Oct 31 01:40:35.557716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.557692 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca\") pod \"35dc0018-4a87-42ce-850d-91e328b9d54b\" (UID: \"35dc0018-4a87-42ce-850d-91e328b9d54b\") " Oct 31 01:40:35.558073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.558050 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca" pod "35dc0018-4a87-42ce-850d-91e328b9d54b" (UID: "35dc0018-4a87-42ce-850d-91e328b9d54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:35.559757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.559709 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca" pod "35dc0018-4a87-42ce-850d-91e328b9d54b" (UID: "35dc0018-4a87-42ce-850d-91e328b9d54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:35.559826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.559755 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk" pod "35dc0018-4a87-42ce-850d-91e328b9d54b" (UID: "35dc0018-4a87-42ce-850d-91e328b9d54b"). InnerVolumeSpecName "kube-api-access-xqfmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:35.658740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.658705 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:35.658740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.658734 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35dc0018-4a87-42ce-850d-91e328b9d54b-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:35.658740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:35.658747 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfmk\" (UniqueName: \"kubernetes.io/projected/35dc0018-4a87-42ce-850d-91e328b9d54b-kube-api-access-xqfmk\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:36.377832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.377804 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" Oct 31 01:40:36.378029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.377809 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l" event={"ID":"35dc0018-4a87-42ce-850d-91e328b9d54b","Type":"ContainerDied","Data":"cf184d627e3a1a4e73e8a49585b473818fccfbb87a7f81584ea77dd843bb6b74"} Oct 31 01:40:36.378029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.377916 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf184d627e3a1a4e73e8a49585b473818fccfbb87a7f81584ea77dd843bb6b74" Oct 31 01:40:36.795549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.795516 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf"] Oct 31 01:40:36.796342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.796317 5756 cpu_manager.go:414] "containerMap: removing container" podUID="35dc0018-4a87-42ce-850d-91e328b9d54b" containerName="revoke-keys" Oct 31 01:40:36.796443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.796342 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35dc0018-4a87-42ce-850d-91e328b9d54b" containerName="revoke-keys" Oct 31 01:40:36.796511 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.796496 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="35dc0018-4a87-42ce-850d-91e328b9d54b" containerName="revoke-keys" Oct 31 01:40:36.800375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.800357 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:36.855272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.855242 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf"] Oct 31 01:40:36.869818 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.869784 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9d7m\" (UniqueName: \"kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m\") pod \"payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:36.870001 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.869888 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:36.901180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.901116 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-66100b\"" Oct 31 01:40:36.901180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.901141 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-66100b-dockercfg-65sfk\"" Oct 31 01:40:36.952626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.952587 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj"] Oct 31 01:40:36.956456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.956432 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:36.980297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.980269 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:36.980477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.980311 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk277\" (UniqueName: \"kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277\") pod \"payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:36.980477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.980338 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9d7m\" (UniqueName: \"kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m\") pod \"payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:36.980595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.980513 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:36.981230 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:36.981208 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:37.001927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.001809 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj"] Oct 31 01:40:37.038247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.038220 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9d7m\" (UniqueName: \"kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m\") pod \"payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:37.081990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.081905 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:37.082161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.082053 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk277\" (UniqueName: \"kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277\") pod \"payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:37.082777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.082751 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:37.113692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.113656 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:37.139518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.139492 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk277\" (UniqueName: \"kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277\") pod \"payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:37.269800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.269767 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:37.502687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.502645 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf"] Oct 31 01:40:37.510196 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:37.510162 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d7af40_120d_4e75_b2b1_63a6e0c5ad6f.slice/crio-a4fab2adebf6460fe1cb4b1238fd5c88568de0054dd3800b6f52f2a2ecab4fe6 WatchSource:0}: Error finding container a4fab2adebf6460fe1cb4b1238fd5c88568de0054dd3800b6f52f2a2ecab4fe6: Status 404 returned error can't find the container with id a4fab2adebf6460fe1cb4b1238fd5c88568de0054dd3800b6f52f2a2ecab4fe6 Oct 31 01:40:37.661564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:37.661537 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj"] Oct 31 01:40:37.664266 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:37.664224 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7620a66e_79fe_40b1_b4aa_56eaf7574e5d.slice/crio-eebe53d8697c489a98d7525c4b25585e931e9b4d7613634ebe36927204cc76b8 WatchSource:0}: Error finding container eebe53d8697c489a98d7525c4b25585e931e9b4d7613634ebe36927204cc76b8: Status 404 returned error can't find the container with id eebe53d8697c489a98d7525c4b25585e931e9b4d7613634ebe36927204cc76b8 Oct 31 01:40:38.391739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.391691 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" event={"ID":"7620a66e-79fe-40b1-b4aa-56eaf7574e5d","Type":"ContainerStarted","Data":"72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf"} Oct 31 01:40:38.392235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.391755 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:38.392235 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.391779 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" event={"ID":"7620a66e-79fe-40b1-b4aa-56eaf7574e5d","Type":"ContainerStarted","Data":"eebe53d8697c489a98d7525c4b25585e931e9b4d7613634ebe36927204cc76b8"} Oct 31 01:40:38.393427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.393403 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" event={"ID":"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f","Type":"ContainerStarted","Data":"6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4"} Oct 31 01:40:38.393515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.393442 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" event={"ID":"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f","Type":"ContainerStarted","Data":"a4fab2adebf6460fe1cb4b1238fd5c88568de0054dd3800b6f52f2a2ecab4fe6"} Oct 31 01:40:38.393515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.393481 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:38.504931 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.504823 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" podStartSLOduration=2.5048044259999998 podStartE2EDuration="2.504804426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:38.504411298 +0000 UTC m=+2610.311378842" watchObservedRunningTime="2026-10-31 01:40:38.504804426 +0000 UTC m=+2610.311771966" Oct 31 01:40:38.771856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:38.771729 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" podStartSLOduration=2.771712806 podStartE2EDuration="2.771712806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:38.769091439 +0000 UTC m=+2610.576058985" watchObservedRunningTime="2026-10-31 01:40:38.771712806 +0000 UTC m=+2610.578680378" Oct 31 01:40:39.979722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:39.979659 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:40:40.016168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016139 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016179 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016231 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016263 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016319 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016370 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016402 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016539 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016562 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:40:40.016606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016577 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:40:40.016606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016588 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:40.016811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016612 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs\") pod \"c70abbbf-469e-495d-bd1f-0101de39b653\" (UID: \"c70abbbf-469e-495d-bd1f-0101de39b653\") " Oct 31 01:40:40.016969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016905 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:40:40.017100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.016969 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:40:40.017100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.017076 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70abbbf-469e-495d-bd1f-0101de39b653-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.017256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.017097 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.017256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.017113 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.017256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.017132 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.017382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.017231 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.018755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.018726 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:40.018903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.018760 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:40:40.019220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.019197 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "kube-api-access-ggncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:40.019220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.019204 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy" pod "c70abbbf-469e-495d-bd1f-0101de39b653" (UID: "c70abbbf-469e-495d-bd1f-0101de39b653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:40:40.119086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.119053 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggncb\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-kube-api-access-ggncb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.119086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.119082 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70abbbf-469e-495d-bd1f-0101de39b653-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.119086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.119091 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70abbbf-469e-495d-bd1f-0101de39b653-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.119314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.119100 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70abbbf-469e-495d-bd1f-0101de39b653-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:40:40.406205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.406173 5756 generic.go:356] "Generic (PLEG): container finished" podID="c70abbbf-469e-495d-bd1f-0101de39b653" containerID="48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d" exitCode=0 Oct 31 01:40:40.406390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.406245 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" Oct 31 01:40:40.406390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.406260 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" event={"ID":"c70abbbf-469e-495d-bd1f-0101de39b653","Type":"ContainerDied","Data":"48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d"} Oct 31 01:40:40.406390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.406338 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp" event={"ID":"c70abbbf-469e-495d-bd1f-0101de39b653","Type":"ContainerDied","Data":"14327b4cb1d6fd20d4fa0837f914482671a58c69aeed44eaa5f241151bc41a2a"} Oct 31 01:40:40.406390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.406360 5756 scope.go:122] "RemoveContainer" containerID="48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d" Oct 31 01:40:40.416133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.416113 5756 scope.go:122] "RemoveContainer" containerID="118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d" Oct 31 01:40:40.425083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.425058 5756 scope.go:122] "RemoveContainer" containerID="48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d" Oct 31 01:40:40.425373 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:40:40.425345 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d\": container with ID starting with 48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d not found: ID does not exist" containerID="48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d" Oct 31 01:40:40.425422 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.425383 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d"} err="failed to get container status \"48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d\": rpc error: code = NotFound desc = could not find container \"48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d\": container with ID starting with 48af01980c16d3fd9451ee704d0932b94b21799886f369c95f1f8f83189e725d not found: ID does not exist" Oct 31 01:40:40.425422 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.425400 5756 scope.go:122] "RemoveContainer" containerID="118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d" Oct 31 01:40:40.425658 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:40:40.425640 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d\": container with ID starting with 118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d not found: ID does not exist" containerID="118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d" Oct 31 01:40:40.425694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.425665 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d"} err="failed to get container status \"118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d\": rpc error: code = NotFound desc = could not find container \"118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d\": container with ID starting with 118b3a0bf6de3bc57009cbec352f98f9bfe07b9ab9a412526e0699f3ed883a2d not found: ID does not exist" Oct 31 01:40:40.568998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.568922 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp"] Oct 31 01:40:40.618318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:40.618288 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3f0818-openshift-default-5b5b7bfbcc-25dfp"] Oct 31 01:40:42.311120 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:42.311080 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" path="/var/lib/kubelet/pods/c70abbbf-469e-495d-bd1f-0101de39b653/volumes" Oct 31 01:40:44.368733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.368703 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:40:44.985637 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.985601 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd"] Oct 31 01:40:44.986169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.986154 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" Oct 31 01:40:44.986212 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.986171 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" Oct 31 01:40:44.986348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.986276 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" Oct 31 01:40:44.986348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.986290 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" Oct 31 01:40:44.990941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:44.990921 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.044540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.044032 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0c0723\"" Oct 31 01:40:45.044540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.044289 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0c0723-dockercfg-j9pbx\"" Oct 31 01:40:45.046954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.046929 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd"] Oct 31 01:40:45.072807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.072776 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x5pr\" (UniqueName: \"kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr\") pod \"payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.073070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.073044 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.149005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.148974 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9"] Oct 31 01:40:45.149524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.149511 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" Oct 31 01:40:45.149572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.149527 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70abbbf-469e-495d-bd1f-0101de39b653" containerName="istio-proxy" Oct 31 01:40:45.152913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.152896 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.173802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.173776 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.173966 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.173826 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x5pr\" (UniqueName: \"kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr\") pod \"payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.174490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.174466 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.208757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.208725 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9"] Oct 31 01:40:45.232453 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.232419 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x5pr\" (UniqueName: \"kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr\") pod \"payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.274986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.274908 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tncl\" (UniqueName: \"kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl\") pod \"payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.274986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.274949 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.304571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.304544 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:45.376904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.376849 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tncl\" (UniqueName: \"kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl\") pod \"payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.377353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.376928 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.378375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.378348 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.434669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.434597 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tncl\" (UniqueName: \"kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl\") pod \"payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.466762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.466730 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:45.679970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.679938 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd"] Oct 31 01:40:45.681493 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:45.681467 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66905eaa_938e_415e_a241_0e29d0ce498a.slice/crio-43009ed8af0470dfd972d5356bff744b37ed629c300886aa96e2c40b330da14a WatchSource:0}: Error finding container 43009ed8af0470dfd972d5356bff744b37ed629c300886aa96e2c40b330da14a: Status 404 returned error can't find the container with id 43009ed8af0470dfd972d5356bff744b37ed629c300886aa96e2c40b330da14a Oct 31 01:40:45.842755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:45.842720 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9"] Oct 31 01:40:45.844561 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:40:45.844537 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0908f8_6f62_468b_9829_f542f699a6c6.slice/crio-b4ccc762765ef8f9b9ba244cedf60788794050783709008bec561d331e1c3380 WatchSource:0}: Error finding container b4ccc762765ef8f9b9ba244cedf60788794050783709008bec561d331e1c3380: Status 404 returned error can't find the container with id b4ccc762765ef8f9b9ba244cedf60788794050783709008bec561d331e1c3380 Oct 31 01:40:46.442804 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.442765 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" event={"ID":"66905eaa-938e-415e-a241-0e29d0ce498a","Type":"ContainerStarted","Data":"27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851"} Oct 31 01:40:46.443254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.442827 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" event={"ID":"66905eaa-938e-415e-a241-0e29d0ce498a","Type":"ContainerStarted","Data":"43009ed8af0470dfd972d5356bff744b37ed629c300886aa96e2c40b330da14a"} Oct 31 01:40:46.443254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.442854 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:40:46.444507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.444484 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" event={"ID":"db0908f8-6f62-468b-9829-f542f699a6c6","Type":"ContainerStarted","Data":"b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f"} Oct 31 01:40:46.444595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.444523 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" event={"ID":"db0908f8-6f62-468b-9829-f542f699a6c6","Type":"ContainerStarted","Data":"b4ccc762765ef8f9b9ba244cedf60788794050783709008bec561d331e1c3380"} Oct 31 01:40:46.444595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.444555 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:46.556375 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.556318 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" podStartSLOduration=2.556305564 podStartE2EDuration="2.556305564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:46.555919085 +0000 UTC m=+2618.362886630" watchObservedRunningTime="2026-10-31 01:40:46.556305564 +0000 UTC m=+2618.363273136" Oct 31 01:40:46.671603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:46.671552 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" podStartSLOduration=1.671538098 podStartE2EDuration="1.671538098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:46.66840618 +0000 UTC m=+2618.475373724" watchObservedRunningTime="2026-10-31 01:40:46.671538098 +0000 UTC m=+2618.478505640" Oct 31 01:40:49.402442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:49.402409 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:40:49.402919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:49.402480 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:40:57.453666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:57.453639 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:40:57.454061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:40:57.453713 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:41:02.066208 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:41:02.066168 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:41:14.511356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:41:14.511325 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:41:28.924429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:41:28.924390 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4"] Oct 31 01:41:28.973302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:41:28.973269 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9da8f-jl4m4"] Oct 31 01:41:30.312044 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:41:30.312014 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1674753c-59af-4074-b3f5-355c772fb44d" path="/var/lib/kubelet/pods/1674753c-59af-4074-b3f5-355c772fb44d/volumes" Oct 31 01:41:32.066527 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:41:32.066493 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:41:59.134418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:41:59.134379 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:42:01.676312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.676280 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr"] Oct 31 01:42:01.680686 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.680660 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.736140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.736093 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr"] Oct 31 01:42:01.748458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.748411 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2bnj\" (UniqueName: \"kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.748643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.748597 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.748724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.748641 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.782788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.782753 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:42:01.782788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.782761 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:42:01.849681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.849646 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2bnj\" (UniqueName: \"kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.849903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.849721 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.849903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.849750 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.850443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.850422 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.852407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.852386 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.908678 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.908649 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2bnj\" (UniqueName: \"kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj\") pod \"maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:01.996270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:01.996193 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:02.074776 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:42:02.074734 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:42:02.370943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:02.370920 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr"] Oct 31 01:42:02.373458 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:42:02.373423 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476ab8f4_fac6_4d05_b7c2_c98595f20e77.slice/crio-65002b55f5676bc0b9008380aa35e55de53375421e211b22fe364a48db003045 WatchSource:0}: Error finding container 65002b55f5676bc0b9008380aa35e55de53375421e211b22fe364a48db003045: Status 404 returned error can't find the container with id 65002b55f5676bc0b9008380aa35e55de53375421e211b22fe364a48db003045 Oct 31 01:42:02.925770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:02.925686 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" event={"ID":"476ab8f4-fac6-4d05-b7c2-c98595f20e77","Type":"ContainerStarted","Data":"5fca679070277536b1d9bf6778c73bad3d40bb622107ea6be10b95214c14594a"} Oct 31 01:42:02.925770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:02.925739 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" event={"ID":"476ab8f4-fac6-4d05-b7c2-c98595f20e77","Type":"ContainerStarted","Data":"65002b55f5676bc0b9008380aa35e55de53375421e211b22fe364a48db003045"} Oct 31 01:42:03.040600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:03.040539 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" podStartSLOduration=2.040524849 podStartE2EDuration="2.040524849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:03.039438512 +0000 UTC m=+2694.846406057" watchObservedRunningTime="2026-10-31 01:42:03.040524849 +0000 UTC m=+2694.847492391" Oct 31 01:42:03.932995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:03.932962 5756 generic.go:356] "Generic (PLEG): container finished" podID="476ab8f4-fac6-4d05-b7c2-c98595f20e77" containerID="5fca679070277536b1d9bf6778c73bad3d40bb622107ea6be10b95214c14594a" exitCode=0 Oct 31 01:42:03.933441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:03.933016 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" event={"ID":"476ab8f4-fac6-4d05-b7c2-c98595f20e77","Type":"ContainerDied","Data":"5fca679070277536b1d9bf6778c73bad3d40bb622107ea6be10b95214c14594a"} Oct 31 01:42:05.086004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.085981 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:05.184320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.184290 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj\" (UniqueName: \"kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj\") pod \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " Oct 31 01:42:05.184494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.184341 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca\") pod \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " Oct 31 01:42:05.184494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.184396 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca\") pod \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\" (UID: \"476ab8f4-fac6-4d05-b7c2-c98595f20e77\") " Oct 31 01:42:05.184756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.184732 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca" pod "476ab8f4-fac6-4d05-b7c2-c98595f20e77" (UID: "476ab8f4-fac6-4d05-b7c2-c98595f20e77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:42:05.186619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.186589 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj" pod "476ab8f4-fac6-4d05-b7c2-c98595f20e77" (UID: "476ab8f4-fac6-4d05-b7c2-c98595f20e77"). InnerVolumeSpecName "kube-api-access-d2bnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:42:05.186763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.186744 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca" pod "476ab8f4-fac6-4d05-b7c2-c98595f20e77" (UID: "476ab8f4-fac6-4d05-b7c2-c98595f20e77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:42:05.285461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.285424 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2bnj\" (UniqueName: \"kubernetes.io/projected/476ab8f4-fac6-4d05-b7c2-c98595f20e77-kube-api-access-d2bnj\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:05.285461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.285452 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:05.285461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.285463 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476ab8f4-fac6-4d05-b7c2-c98595f20e77-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:05.946581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.946546 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" event={"ID":"476ab8f4-fac6-4d05-b7c2-c98595f20e77","Type":"ContainerDied","Data":"65002b55f5676bc0b9008380aa35e55de53375421e211b22fe364a48db003045"} Oct 31 01:42:05.946581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.946580 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr" Oct 31 01:42:05.946798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:05.946595 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65002b55f5676bc0b9008380aa35e55de53375421e211b22fe364a48db003045" Oct 31 01:42:27.525365 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.525330 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk"] Oct 31 01:42:27.527649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.525806 5756 cpu_manager.go:414] "containerMap: removing container" podUID="476ab8f4-fac6-4d05-b7c2-c98595f20e77" containerName="revoke-keys" Oct 31 01:42:27.527649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.525816 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476ab8f4-fac6-4d05-b7c2-c98595f20e77" containerName="revoke-keys" Oct 31 01:42:27.527649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.525949 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="476ab8f4-fac6-4d05-b7c2-c98595f20e77" containerName="revoke-keys" Oct 31 01:42:27.529100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.529085 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.632035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.631990 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.632231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.632093 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.632231 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.632202 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn8rb\" (UniqueName: \"kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.632393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.632376 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:42:27.632505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.632488 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:42:27.634482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.634463 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk"] Oct 31 01:42:27.733800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.733764 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.734020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.733812 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.734020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.733970 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn8rb\" (UniqueName: \"kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.734537 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.734516 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.736302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.736284 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.791011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.790978 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn8rb\" (UniqueName: \"kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb\") pod \"maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:27.841906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:27.841854 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:28.218618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:28.218591 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk"] Oct 31 01:42:28.220583 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:42:28.220553 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901ef041_ab0c_42ef_817e_ad2873d357c0.slice/crio-063afc39e7f570ffba6ef68936df1268b8dd369a2117136f472e2b5d3d3a108c WatchSource:0}: Error finding container 063afc39e7f570ffba6ef68936df1268b8dd369a2117136f472e2b5d3d3a108c: Status 404 returned error can't find the container with id 063afc39e7f570ffba6ef68936df1268b8dd369a2117136f472e2b5d3d3a108c Oct 31 01:42:29.086706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:29.086667 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" event={"ID":"901ef041-ab0c-42ef-817e-ad2873d357c0","Type":"ContainerStarted","Data":"de3f4490a1cb38f6cc30ea9cf98d4af5f36d19cfcf00893a21a7596e6a98f98b"} Oct 31 01:42:29.087175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:29.086736 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" event={"ID":"901ef041-ab0c-42ef-817e-ad2873d357c0","Type":"ContainerStarted","Data":"063afc39e7f570ffba6ef68936df1268b8dd369a2117136f472e2b5d3d3a108c"} Oct 31 01:42:29.200533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:29.200476 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" podStartSLOduration=2.200461631 podStartE2EDuration="2.200461631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:29.199052979 +0000 UTC m=+2721.006020525" watchObservedRunningTime="2026-10-31 01:42:29.200461631 +0000 UTC m=+2721.007429175" Oct 31 01:42:30.094149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:30.094113 5756 generic.go:356] "Generic (PLEG): container finished" podID="901ef041-ab0c-42ef-817e-ad2873d357c0" containerID="de3f4490a1cb38f6cc30ea9cf98d4af5f36d19cfcf00893a21a7596e6a98f98b" exitCode=0 Oct 31 01:42:30.094545 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:30.094171 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" event={"ID":"901ef041-ab0c-42ef-817e-ad2873d357c0","Type":"ContainerDied","Data":"de3f4490a1cb38f6cc30ea9cf98d4af5f36d19cfcf00893a21a7596e6a98f98b"} Oct 31 01:42:31.240245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.240221 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:31.377609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.377534 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb\" (UniqueName: \"kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb\") pod \"901ef041-ab0c-42ef-817e-ad2873d357c0\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " Oct 31 01:42:31.377747 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.377625 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca\") pod \"901ef041-ab0c-42ef-817e-ad2873d357c0\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " Oct 31 01:42:31.377747 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.377672 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca\") pod \"901ef041-ab0c-42ef-817e-ad2873d357c0\" (UID: \"901ef041-ab0c-42ef-817e-ad2873d357c0\") " Oct 31 01:42:31.378136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.378109 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca" pod "901ef041-ab0c-42ef-817e-ad2873d357c0" (UID: "901ef041-ab0c-42ef-817e-ad2873d357c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:42:31.379928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.379897 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb" pod "901ef041-ab0c-42ef-817e-ad2873d357c0" (UID: "901ef041-ab0c-42ef-817e-ad2873d357c0"). InnerVolumeSpecName "kube-api-access-fn8rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:42:31.380050 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.379995 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca" pod "901ef041-ab0c-42ef-817e-ad2873d357c0" (UID: "901ef041-ab0c-42ef-817e-ad2873d357c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:42:31.479847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.479808 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:31.479847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.479848 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901ef041-ab0c-42ef-817e-ad2873d357c0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:31.480080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:31.479895 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn8rb\" (UniqueName: \"kubernetes.io/projected/901ef041-ab0c-42ef-817e-ad2873d357c0-kube-api-access-fn8rb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:32.067146 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:42:32.067112 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:42:32.107827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:32.107791 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" event={"ID":"901ef041-ab0c-42ef-817e-ad2873d357c0","Type":"ContainerDied","Data":"063afc39e7f570ffba6ef68936df1268b8dd369a2117136f472e2b5d3d3a108c"} Oct 31 01:42:32.107999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:32.107845 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="063afc39e7f570ffba6ef68936df1268b8dd369a2117136f472e2b5d3d3a108c" Oct 31 01:42:32.107999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:32.107801 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk" Oct 31 01:42:39.028498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:39.028464 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv"] Oct 31 01:42:43.396324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.396284 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74"] Oct 31 01:42:43.397040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.397019 5756 cpu_manager.go:414] "containerMap: removing container" podUID="901ef041-ab0c-42ef-817e-ad2873d357c0" containerName="revoke-keys" Oct 31 01:42:43.397153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.397041 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901ef041-ab0c-42ef-817e-ad2873d357c0" containerName="revoke-keys" Oct 31 01:42:43.397213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.397193 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="901ef041-ab0c-42ef-817e-ad2873d357c0" containerName="revoke-keys" Oct 31 01:42:43.400614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.400591 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.502970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.502932 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ade881\"" Oct 31 01:42:43.503137 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.502976 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ade881-dockercfg-2dnrq\"" Oct 31 01:42:43.505286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.505265 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74"] Oct 31 01:42:43.506474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.506446 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.506583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.506515 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbsn\" (UniqueName: \"kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn\") pod \"payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.607336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.607298 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.607498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.607361 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbsn\" (UniqueName: \"kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn\") pod \"payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.608055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.608037 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.664665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.664607 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbsn\" (UniqueName: \"kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn\") pod \"payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:43.714685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:43.714659 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:44.029361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.029249 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" podUID="362d427b-b6d5-457b-9a01-0648274ff655" containerName="payload-processing" containerID="cri-o://37ca05d0abed4378a8593907521af437816b2ba94d9cd36fa2a57ee02e669443" gracePeriod=25 Oct 31 01:42:44.093728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.092766 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74"] Oct 31 01:42:44.111094 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:42:44.111061 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd2808f_9e8e_44d2_ab6e_3eaab892a35b.slice/crio-e02e9a8868f94d0027d1a702b06a97814652085b3116868005e605c7863ae3c0 WatchSource:0}: Error finding container e02e9a8868f94d0027d1a702b06a97814652085b3116868005e605c7863ae3c0: Status 404 returned error can't find the container with id e02e9a8868f94d0027d1a702b06a97814652085b3116868005e605c7863ae3c0 Oct 31 01:42:44.180658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.180619 5756 generic.go:356] "Generic (PLEG): container finished" podID="362d427b-b6d5-457b-9a01-0648274ff655" containerID="37ca05d0abed4378a8593907521af437816b2ba94d9cd36fa2a57ee02e669443" exitCode=0 Oct 31 01:42:44.180802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.180678 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" event={"ID":"362d427b-b6d5-457b-9a01-0648274ff655","Type":"ContainerDied","Data":"37ca05d0abed4378a8593907521af437816b2ba94d9cd36fa2a57ee02e669443"} Oct 31 01:42:44.182317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.182288 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" event={"ID":"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b","Type":"ContainerStarted","Data":"e02e9a8868f94d0027d1a702b06a97814652085b3116868005e605c7863ae3c0"} Oct 31 01:42:44.297711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.297683 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:42:44.419888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.418962 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume\") pod \"362d427b-b6d5-457b-9a01-0648274ff655\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " Oct 31 01:42:44.419888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.419130 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb\" (UniqueName: \"kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb\") pod \"362d427b-b6d5-457b-9a01-0648274ff655\" (UID: \"362d427b-b6d5-457b-9a01-0648274ff655\") " Oct 31 01:42:44.419888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.419443 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume" pod "362d427b-b6d5-457b-9a01-0648274ff655" (UID: "362d427b-b6d5-457b-9a01-0648274ff655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:42:44.419888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.419600 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/362d427b-b6d5-457b-9a01-0648274ff655-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:44.422290 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.422262 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb" pod "362d427b-b6d5-457b-9a01-0648274ff655" (UID: "362d427b-b6d5-457b-9a01-0648274ff655"). InnerVolumeSpecName "kube-api-access-s8jfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:42:44.521101 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:44.521070 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jfb\" (UniqueName: \"kubernetes.io/projected/362d427b-b6d5-457b-9a01-0648274ff655-kube-api-access-s8jfb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:45.192362 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.192328 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" Oct 31 01:42:45.192602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.192363 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv" event={"ID":"362d427b-b6d5-457b-9a01-0648274ff655","Type":"ContainerDied","Data":"e40902da4544583b0a39c4511570c853ef18e85ed89dd77fcd2c0ee5c7469296"} Oct 31 01:42:45.192602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.192472 5756 scope.go:122] "RemoveContainer" containerID="37ca05d0abed4378a8593907521af437816b2ba94d9cd36fa2a57ee02e669443" Oct 31 01:42:45.194347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.194319 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" event={"ID":"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b","Type":"ContainerStarted","Data":"6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb"} Oct 31 01:42:45.195095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.195075 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:45.361685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.361631 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" podStartSLOduration=2.361617361 podStartE2EDuration="2.361617361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:45.359994107 +0000 UTC m=+2737.166961647" watchObservedRunningTime="2026-10-31 01:42:45.361617361 +0000 UTC m=+2737.168584923" Oct 31 01:42:45.469810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.469730 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv"] Oct 31 01:42:45.474283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:45.474257 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a9710-68dc7c4d55-wptrv"] Oct 31 01:42:46.311517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:46.311487 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362d427b-b6d5-457b-9a01-0648274ff655" path="/var/lib/kubelet/pods/362d427b-b6d5-457b-9a01-0648274ff655/volumes" Oct 31 01:42:46.457595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:46.457557 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr"] Oct 31 01:42:46.506771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:46.506741 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a31828-7hrcr"] Oct 31 01:42:47.561929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:47.561899 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk"] Oct 31 01:42:47.611446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:47.611418 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6a9710-5ngvk"] Oct 31 01:42:48.310878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:48.310841 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476ab8f4-fac6-4d05-b7c2-c98595f20e77" path="/var/lib/kubelet/pods/476ab8f4-fac6-4d05-b7c2-c98595f20e77/volumes" Oct 31 01:42:48.311307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:48.311293 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901ef041-ab0c-42ef-817e-ad2873d357c0" path="/var/lib/kubelet/pods/901ef041-ab0c-42ef-817e-ad2873d357c0/volumes" Oct 31 01:42:55.616279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.616247 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq"] Oct 31 01:42:55.616774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.616760 5756 cpu_manager.go:414] "containerMap: removing container" podUID="362d427b-b6d5-457b-9a01-0648274ff655" containerName="payload-processing" Oct 31 01:42:55.616823 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.616774 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362d427b-b6d5-457b-9a01-0648274ff655" containerName="payload-processing" Oct 31 01:42:55.616890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.616873 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="362d427b-b6d5-457b-9a01-0648274ff655" containerName="payload-processing" Oct 31 01:42:55.621760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.621742 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.676371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.676337 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq"] Oct 31 01:42:55.722553 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.722525 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:42:55.722722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.722577 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:42:55.747505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.747478 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czb55\" (UniqueName: \"kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.747672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.747550 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.747672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.747650 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.848583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.848553 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.848779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.848640 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czb55\" (UniqueName: \"kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.848779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.848702 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.849345 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.849326 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.851104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.851080 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.905509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.905441 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czb55\" (UniqueName: \"kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55\") pod \"maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:55.935562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:55.935540 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:56.309396 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:42:56.309368 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6b34b5_b21f_4563_bed3_5fe38abb4133.slice/crio-685abc564eddbb1e4cc4591c2ff7fdfb65aca81c2efcd7145aef589644552170 WatchSource:0}: Error finding container 685abc564eddbb1e4cc4591c2ff7fdfb65aca81c2efcd7145aef589644552170: Status 404 returned error can't find the container with id 685abc564eddbb1e4cc4591c2ff7fdfb65aca81c2efcd7145aef589644552170 Oct 31 01:42:56.312411 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:56.312388 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq"] Oct 31 01:42:57.210397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:57.210365 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:42:57.277057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:57.277030 5756 generic.go:356] "Generic (PLEG): container finished" podID="aa6b34b5-b21f-4563-bed3-5fe38abb4133" containerID="392a0c04b79c1b53af92ce0cd3b090b17fd0bdfed68f7adbe15915395680b849" exitCode=0 Oct 31 01:42:57.277221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:57.277110 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" event={"ID":"aa6b34b5-b21f-4563-bed3-5fe38abb4133","Type":"ContainerDied","Data":"392a0c04b79c1b53af92ce0cd3b090b17fd0bdfed68f7adbe15915395680b849"} Oct 31 01:42:57.277221 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:57.277171 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" event={"ID":"aa6b34b5-b21f-4563-bed3-5fe38abb4133","Type":"ContainerStarted","Data":"685abc564eddbb1e4cc4591c2ff7fdfb65aca81c2efcd7145aef589644552170"} Oct 31 01:42:58.423281 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.423259 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:58.581797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.581716 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca\") pod \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " Oct 31 01:42:58.581981 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.581932 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55\" (UniqueName: \"kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55\") pod \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " Oct 31 01:42:58.582050 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.582015 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca\") pod \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\" (UID: \"aa6b34b5-b21f-4563-bed3-5fe38abb4133\") " Oct 31 01:42:58.582359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.582338 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca" pod "aa6b34b5-b21f-4563-bed3-5fe38abb4133" (UID: "aa6b34b5-b21f-4563-bed3-5fe38abb4133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:42:58.583907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.583883 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca" pod "aa6b34b5-b21f-4563-bed3-5fe38abb4133" (UID: "aa6b34b5-b21f-4563-bed3-5fe38abb4133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:42:58.584178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.584158 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55" pod "aa6b34b5-b21f-4563-bed3-5fe38abb4133" (UID: "aa6b34b5-b21f-4563-bed3-5fe38abb4133"). InnerVolumeSpecName "kube-api-access-czb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:42:58.684123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.684093 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czb55\" (UniqueName: \"kubernetes.io/projected/aa6b34b5-b21f-4563-bed3-5fe38abb4133-kube-api-access-czb55\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:58.684123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.684118 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:58.684123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:58.684128 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6b34b5-b21f-4563-bed3-5fe38abb4133-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:42:59.291465 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:59.291438 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" Oct 31 01:42:59.291694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:59.291511 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq" event={"ID":"aa6b34b5-b21f-4563-bed3-5fe38abb4133","Type":"ContainerDied","Data":"685abc564eddbb1e4cc4591c2ff7fdfb65aca81c2efcd7145aef589644552170"} Oct 31 01:42:59.291694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:42:59.291577 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="685abc564eddbb1e4cc4591c2ff7fdfb65aca81c2efcd7145aef589644552170" Oct 31 01:43:01.593043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:01.593005 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:43:02.066058 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:43:02.066023 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:43:07.228171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:07.228132 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w"] Oct 31 01:43:10.026179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:10.026149 5756 scope.go:122] "RemoveContainer" containerID="4ea5597cf2412766c3c3a78a4c46285c73a21b753d9fe4ebbcc10edf28fa8c42" Oct 31 01:43:10.037255 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:10.037238 5756 scope.go:122] "RemoveContainer" containerID="dce1c52722cd2f862bd6e06663c54a79cdb2ad77fdfbe61fc00ce3961d6ed413" Oct 31 01:43:12.228538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.228478 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" podUID="76ac59c2-43f5-484f-914c-c9edcbfaf0ad" containerName="payload-pre-processing" containerID="cri-o://269c2ed0ce0da14c9958a76a7c3af3b8fe2cc4d0a66f73f8e1975510ec899796" gracePeriod=25 Oct 31 01:43:12.376573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.376532 5756 generic.go:356] "Generic (PLEG): container finished" podID="76ac59c2-43f5-484f-914c-c9edcbfaf0ad" containerID="269c2ed0ce0da14c9958a76a7c3af3b8fe2cc4d0a66f73f8e1975510ec899796" exitCode=0 Oct 31 01:43:12.376753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.376605 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" event={"ID":"76ac59c2-43f5-484f-914c-c9edcbfaf0ad","Type":"ContainerDied","Data":"269c2ed0ce0da14c9958a76a7c3af3b8fe2cc4d0a66f73f8e1975510ec899796"} Oct 31 01:43:12.481910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.481824 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:43:12.537730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.537697 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume\") pod \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " Oct 31 01:43:12.537924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.537908 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72\" (UniqueName: \"kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72\") pod \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\" (UID: \"76ac59c2-43f5-484f-914c-c9edcbfaf0ad\") " Oct 31 01:43:12.538155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.538134 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume" pod "76ac59c2-43f5-484f-914c-c9edcbfaf0ad" (UID: "76ac59c2-43f5-484f-914c-c9edcbfaf0ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:43:12.538309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.538294 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:12.540103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.540080 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72" pod "76ac59c2-43f5-484f-914c-c9edcbfaf0ad" (UID: "76ac59c2-43f5-484f-914c-c9edcbfaf0ad"). InnerVolumeSpecName "kube-api-access-8qn72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:43:12.639198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:12.639157 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qn72\" (UniqueName: \"kubernetes.io/projected/76ac59c2-43f5-484f-914c-c9edcbfaf0ad-kube-api-access-8qn72\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:13.384127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:13.384097 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" Oct 31 01:43:13.384127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:13.384112 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w" event={"ID":"76ac59c2-43f5-484f-914c-c9edcbfaf0ad","Type":"ContainerDied","Data":"37e2b22404a3514c94ef0c51c082ac0069cb3d8b23c9f9e49ebd8d6089c1e929"} Oct 31 01:43:13.384607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:13.384184 5756 scope.go:122] "RemoveContainer" containerID="269c2ed0ce0da14c9958a76a7c3af3b8fe2cc4d0a66f73f8e1975510ec899796" Oct 31 01:43:13.547115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:13.547084 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w"] Oct 31 01:43:13.596700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:13.596663 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d867bb-68c8c75675-hd65w"] Oct 31 01:43:14.311554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:14.311524 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ac59c2-43f5-484f-914c-c9edcbfaf0ad" path="/var/lib/kubelet/pods/76ac59c2-43f5-484f-914c-c9edcbfaf0ad/volumes" Oct 31 01:43:14.586018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:14.585885 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq"] Oct 31 01:43:14.635227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:14.635199 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-d867bb-lc7gq"] Oct 31 01:43:16.312013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:16.311968 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6b34b5-b21f-4563-bed3-5fe38abb4133" path="/var/lib/kubelet/pods/aa6b34b5-b21f-4563-bed3-5fe38abb4133/volumes" Oct 31 01:43:32.065934 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:43:32.065896 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:43:32.707953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.707916 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c"] Oct 31 01:43:32.708499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.708478 5756 cpu_manager.go:414] "containerMap: removing container" podUID="aa6b34b5-b21f-4563-bed3-5fe38abb4133" containerName="revoke-keys" Oct 31 01:43:32.708598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.708500 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6b34b5-b21f-4563-bed3-5fe38abb4133" containerName="revoke-keys" Oct 31 01:43:32.708598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.708551 5756 cpu_manager.go:414] "containerMap: removing container" podUID="76ac59c2-43f5-484f-914c-c9edcbfaf0ad" containerName="payload-pre-processing" Oct 31 01:43:32.708598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.708560 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ac59c2-43f5-484f-914c-c9edcbfaf0ad" containerName="payload-pre-processing" Oct 31 01:43:32.708758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.708718 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6b34b5-b21f-4563-bed3-5fe38abb4133" containerName="revoke-keys" Oct 31 01:43:32.708758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.708739 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ac59c2-43f5-484f-914c-c9edcbfaf0ad" containerName="payload-pre-processing" Oct 31 01:43:32.713734 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.713712 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.815938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.815903 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:43:32.815938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.815940 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:43:32.818169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.818148 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c"] Oct 31 01:43:32.857891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.857841 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.858062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.857908 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.858062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.858028 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtnbw\" (UniqueName: \"kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.959141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.959064 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtnbw\" (UniqueName: \"kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.959287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.959206 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.959287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.959226 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.959923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.959893 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:32.961640 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:32.961619 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:33.016764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:33.016730 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtnbw\" (UniqueName: \"kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw\") pod \"maas-api-revoke-sub-8244c7e5-7rd4c\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:33.027303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:33.027277 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:33.407038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:33.406995 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c"] Oct 31 01:43:33.410450 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:43:33.410397 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3169a51_7385_4060_ab3d_6e8e920f2d61.slice/crio-a2774bcd35baa3ef6c83261b39f3b1381ab36849ca43c989fee23a9004b2874f WatchSource:0}: Error finding container a2774bcd35baa3ef6c83261b39f3b1381ab36849ca43c989fee23a9004b2874f: Status 404 returned error can't find the container with id a2774bcd35baa3ef6c83261b39f3b1381ab36849ca43c989fee23a9004b2874f Oct 31 01:43:33.505318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:33.505282 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" event={"ID":"d3169a51-7385-4060-ab3d-6e8e920f2d61","Type":"ContainerStarted","Data":"a2774bcd35baa3ef6c83261b39f3b1381ab36849ca43c989fee23a9004b2874f"} Oct 31 01:43:34.513090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:34.513050 5756 generic.go:356] "Generic (PLEG): container finished" podID="d3169a51-7385-4060-ab3d-6e8e920f2d61" containerID="44a2f4c68e5ba1d200cb7a46a439886818016f2b8c6fc3749bf8e6ac025e07e2" exitCode=0 Oct 31 01:43:34.513503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:34.513141 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" event={"ID":"d3169a51-7385-4060-ab3d-6e8e920f2d61","Type":"ContainerDied","Data":"44a2f4c68e5ba1d200cb7a46a439886818016f2b8c6fc3749bf8e6ac025e07e2"} Oct 31 01:43:35.661928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.661903 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:35.794001 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.793971 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca\") pod \"d3169a51-7385-4060-ab3d-6e8e920f2d61\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " Oct 31 01:43:35.794202 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.794076 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca\") pod \"d3169a51-7385-4060-ab3d-6e8e920f2d61\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " Oct 31 01:43:35.794202 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.794105 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw\" (UniqueName: \"kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw\") pod \"d3169a51-7385-4060-ab3d-6e8e920f2d61\" (UID: \"d3169a51-7385-4060-ab3d-6e8e920f2d61\") " Oct 31 01:43:35.794408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.794380 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca" pod "d3169a51-7385-4060-ab3d-6e8e920f2d61" (UID: "d3169a51-7385-4060-ab3d-6e8e920f2d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:43:35.794704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.794683 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:35.796417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.796394 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca" pod "d3169a51-7385-4060-ab3d-6e8e920f2d61" (UID: "d3169a51-7385-4060-ab3d-6e8e920f2d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:43:35.796530 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.796453 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw" pod "d3169a51-7385-4060-ab3d-6e8e920f2d61" (UID: "d3169a51-7385-4060-ab3d-6e8e920f2d61"). InnerVolumeSpecName "kube-api-access-rtnbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:43:35.895616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.895586 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3169a51-7385-4060-ab3d-6e8e920f2d61-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:35.895616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:35.895616 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtnbw\" (UniqueName: \"kubernetes.io/projected/d3169a51-7385-4060-ab3d-6e8e920f2d61-kube-api-access-rtnbw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.526969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:36.526937 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" event={"ID":"d3169a51-7385-4060-ab3d-6e8e920f2d61","Type":"ContainerDied","Data":"a2774bcd35baa3ef6c83261b39f3b1381ab36849ca43c989fee23a9004b2874f"} Oct 31 01:43:36.526969 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:36.526968 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c" Oct 31 01:43:36.527177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:36.526985 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2774bcd35baa3ef6c83261b39f3b1381ab36849ca43c989fee23a9004b2874f" Oct 31 01:43:40.140547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.140512 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj"] Oct 31 01:43:40.141161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.141143 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d3169a51-7385-4060-ab3d-6e8e920f2d61" containerName="revoke-keys" Oct 31 01:43:40.141209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.141163 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3169a51-7385-4060-ab3d-6e8e920f2d61" containerName="revoke-keys" Oct 31 01:43:40.141257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.141249 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3169a51-7385-4060-ab3d-6e8e920f2d61" containerName="revoke-keys" Oct 31 01:43:40.144513 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.144497 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.237421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.237389 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.237593 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.237576 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.237649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.237634 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm69x\" (UniqueName: \"kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.245966 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.245948 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:43:40.246098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.246084 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:43:40.248143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.248125 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj"] Oct 31 01:43:40.338418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.338366 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm69x\" (UniqueName: \"kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.338606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.338443 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.338703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.338685 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.339118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.339098 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.341170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.341150 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.398158 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.398099 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm69x\" (UniqueName: \"kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x\") pod \"maas-api-revoke-sub-2f7ccbea-5trwj\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.457529 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.457496 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:40.830714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:40.830686 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj"] Oct 31 01:43:40.832230 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:43:40.832207 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e10483_22b9_48b8_9d7d_111458d3cc37.slice/crio-5b0c89f13df5fd6e50d2866d3803e50cefcfd6e586cee3ba10f8fc3212bd10d6 WatchSource:0}: Error finding container 5b0c89f13df5fd6e50d2866d3803e50cefcfd6e586cee3ba10f8fc3212bd10d6: Status 404 returned error can't find the container with id 5b0c89f13df5fd6e50d2866d3803e50cefcfd6e586cee3ba10f8fc3212bd10d6 Oct 31 01:43:41.557556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:41.557515 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" event={"ID":"65e10483-22b9-48b8-9d7d-111458d3cc37","Type":"ContainerStarted","Data":"96b719d028730a578c1516234c71921c1fe7aec8b511fc9d8bcd06fec59f51cb"} Oct 31 01:43:41.558055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:41.557582 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" event={"ID":"65e10483-22b9-48b8-9d7d-111458d3cc37","Type":"ContainerStarted","Data":"5b0c89f13df5fd6e50d2866d3803e50cefcfd6e586cee3ba10f8fc3212bd10d6"} Oct 31 01:43:41.677828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:41.677778 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" podStartSLOduration=1.677763799 podStartE2EDuration="1.677763799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:41.676494468 +0000 UTC m=+2793.483462014" watchObservedRunningTime="2026-10-31 01:43:41.677763799 +0000 UTC m=+2793.484731341" Oct 31 01:43:42.569289 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:42.569252 5756 generic.go:356] "Generic (PLEG): container finished" podID="65e10483-22b9-48b8-9d7d-111458d3cc37" containerID="96b719d028730a578c1516234c71921c1fe7aec8b511fc9d8bcd06fec59f51cb" exitCode=0 Oct 31 01:43:42.569682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:42.569338 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" event={"ID":"65e10483-22b9-48b8-9d7d-111458d3cc37","Type":"ContainerDied","Data":"96b719d028730a578c1516234c71921c1fe7aec8b511fc9d8bcd06fec59f51cb"} Oct 31 01:43:43.057962 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.057931 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:43:43.756880 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.756840 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:43.774656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.774621 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x\" (UniqueName: \"kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x\") pod \"65e10483-22b9-48b8-9d7d-111458d3cc37\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " Oct 31 01:43:43.774836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.774776 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca\") pod \"65e10483-22b9-48b8-9d7d-111458d3cc37\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " Oct 31 01:43:43.774836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.774814 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca\") pod \"65e10483-22b9-48b8-9d7d-111458d3cc37\" (UID: \"65e10483-22b9-48b8-9d7d-111458d3cc37\") " Oct 31 01:43:43.775237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.775213 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca" pod "65e10483-22b9-48b8-9d7d-111458d3cc37" (UID: "65e10483-22b9-48b8-9d7d-111458d3cc37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:43:43.776844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.776814 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x" pod "65e10483-22b9-48b8-9d7d-111458d3cc37" (UID: "65e10483-22b9-48b8-9d7d-111458d3cc37"). InnerVolumeSpecName "kube-api-access-gm69x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:43:43.777164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.777145 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca" pod "65e10483-22b9-48b8-9d7d-111458d3cc37" (UID: "65e10483-22b9-48b8-9d7d-111458d3cc37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:43:43.876776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.876703 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm69x\" (UniqueName: \"kubernetes.io/projected/65e10483-22b9-48b8-9d7d-111458d3cc37-kube-api-access-gm69x\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:43.876776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.876732 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:43.876776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:43.876742 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65e10483-22b9-48b8-9d7d-111458d3cc37-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:44.595911 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:44.595809 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" Oct 31 01:43:44.595911 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:44.595805 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj" event={"ID":"65e10483-22b9-48b8-9d7d-111458d3cc37","Type":"ContainerDied","Data":"5b0c89f13df5fd6e50d2866d3803e50cefcfd6e586cee3ba10f8fc3212bd10d6"} Oct 31 01:43:44.596132 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:44.595928 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b0c89f13df5fd6e50d2866d3803e50cefcfd6e586cee3ba10f8fc3212bd10d6" Oct 31 01:43:48.436713 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.436625 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx"] Oct 31 01:43:48.437371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.437351 5756 cpu_manager.go:414] "containerMap: removing container" podUID="65e10483-22b9-48b8-9d7d-111458d3cc37" containerName="revoke-keys" Oct 31 01:43:48.437482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.437374 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65e10483-22b9-48b8-9d7d-111458d3cc37" containerName="revoke-keys" Oct 31 01:43:48.437566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.437552 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="65e10483-22b9-48b8-9d7d-111458d3cc37" containerName="revoke-keys" Oct 31 01:43:48.441003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.440984 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.513180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.513148 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx"] Oct 31 01:43:48.533305 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.533264 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.533492 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.533392 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq6k7\" (UniqueName: \"kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.533660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.533640 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.553124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.553086 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:43:48.553124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.553124 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:43:48.634541 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.634505 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.634726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.634589 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.634726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.634633 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq6k7\" (UniqueName: \"kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.635153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.635135 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.637283 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.637260 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.692048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.691980 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq6k7\" (UniqueName: \"kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7\") pod \"maas-api-revoke-sub-c4f83914-2qxnx\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:48.754137 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:48.754109 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:49.133652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:49.133622 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx"] Oct 31 01:43:49.135363 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:43:49.135330 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bd79ba_3d07_4b8c_9991_da320d0d68c7.slice/crio-f09c356b7353df278d31acb4fd888f1fe6e2ca70a46a75fb4389acbd7d59546d WatchSource:0}: Error finding container f09c356b7353df278d31acb4fd888f1fe6e2ca70a46a75fb4389acbd7d59546d: Status 404 returned error can't find the container with id f09c356b7353df278d31acb4fd888f1fe6e2ca70a46a75fb4389acbd7d59546d Oct 31 01:43:49.625769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:49.625735 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" event={"ID":"d0bd79ba-3d07-4b8c-9991-da320d0d68c7","Type":"ContainerStarted","Data":"1e02fe11391adb2f1764efca30a970ce9418bf6435ed16862b9cd5f13b42278b"} Oct 31 01:43:49.626201 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:49.625792 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" event={"ID":"d0bd79ba-3d07-4b8c-9991-da320d0d68c7","Type":"ContainerStarted","Data":"f09c356b7353df278d31acb4fd888f1fe6e2ca70a46a75fb4389acbd7d59546d"} Oct 31 01:43:49.746369 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:49.746273 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" podStartSLOduration=1.746259811 podStartE2EDuration="1.746259811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:49.743628843 +0000 UTC m=+2801.550596387" watchObservedRunningTime="2026-10-31 01:43:49.746259811 +0000 UTC m=+2801.553227354" Oct 31 01:43:50.633564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:50.633528 5756 generic.go:356] "Generic (PLEG): container finished" podID="d0bd79ba-3d07-4b8c-9991-da320d0d68c7" containerID="1e02fe11391adb2f1764efca30a970ce9418bf6435ed16862b9cd5f13b42278b" exitCode=0 Oct 31 01:43:50.633990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:50.633587 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" event={"ID":"d0bd79ba-3d07-4b8c-9991-da320d0d68c7","Type":"ContainerDied","Data":"1e02fe11391adb2f1764efca30a970ce9418bf6435ed16862b9cd5f13b42278b"} Oct 31 01:43:51.790383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.790346 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:51.872698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.872662 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca\") pod \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " Oct 31 01:43:51.872898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.872760 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7\" (UniqueName: \"kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7\") pod \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " Oct 31 01:43:51.872898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.872787 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca\") pod \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\" (UID: \"d0bd79ba-3d07-4b8c-9991-da320d0d68c7\") " Oct 31 01:43:51.873022 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.873002 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca" pod "d0bd79ba-3d07-4b8c-9991-da320d0d68c7" (UID: "d0bd79ba-3d07-4b8c-9991-da320d0d68c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:43:51.873209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.873194 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:51.874998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.874976 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7" pod "d0bd79ba-3d07-4b8c-9991-da320d0d68c7" (UID: "d0bd79ba-3d07-4b8c-9991-da320d0d68c7"). InnerVolumeSpecName "kube-api-access-dq6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:43:51.875241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.875137 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca" pod "d0bd79ba-3d07-4b8c-9991-da320d0d68c7" (UID: "d0bd79ba-3d07-4b8c-9991-da320d0d68c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:43:51.974530 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.974455 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq6k7\" (UniqueName: \"kubernetes.io/projected/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-kube-api-access-dq6k7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:51.974530 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:51.974480 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0bd79ba-3d07-4b8c-9991-da320d0d68c7-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:52.646428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:52.646340 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" event={"ID":"d0bd79ba-3d07-4b8c-9991-da320d0d68c7","Type":"ContainerDied","Data":"f09c356b7353df278d31acb4fd888f1fe6e2ca70a46a75fb4389acbd7d59546d"} Oct 31 01:43:52.646428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:52.646398 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f09c356b7353df278d31acb4fd888f1fe6e2ca70a46a75fb4389acbd7d59546d" Oct 31 01:43:52.646428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:52.646352 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx" Oct 31 01:43:56.616792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.616756 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8"] Oct 31 01:43:56.617421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.617403 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d0bd79ba-3d07-4b8c-9991-da320d0d68c7" containerName="revoke-keys" Oct 31 01:43:56.617494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.617426 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bd79ba-3d07-4b8c-9991-da320d0d68c7" containerName="revoke-keys" Oct 31 01:43:56.617592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.617578 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bd79ba-3d07-4b8c-9991-da320d0d68c7" containerName="revoke-keys" Oct 31 01:43:56.620734 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.620710 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.721898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.721848 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:43:56.722048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.721916 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:43:56.724234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.724215 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8"] Oct 31 01:43:56.726558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.726540 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkwg9\" (UniqueName: \"kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.726606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.726567 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.726770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.726749 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.828435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.828400 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.828645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.828508 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkwg9\" (UniqueName: \"kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.828645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.828539 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.829372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.829341 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.831029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.831007 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.885585 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.885512 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkwg9\" (UniqueName: \"kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9\") pod \"maas-api-revoke-sub-5fd8bea5-vkqz8\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:56.933606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:56.933574 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:57.310305 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:57.310283 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8"] Oct 31 01:43:57.312370 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:43:57.312341 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b376785_4691_42e6_a35d_14bb5de40dd5.slice/crio-e69f2aa64a97f70625458bbd53f8956d54f1c46d9311f8839fdd3e6d6beccfca WatchSource:0}: Error finding container e69f2aa64a97f70625458bbd53f8956d54f1c46d9311f8839fdd3e6d6beccfca: Status 404 returned error can't find the container with id e69f2aa64a97f70625458bbd53f8956d54f1c46d9311f8839fdd3e6d6beccfca Oct 31 01:43:57.677785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:57.677750 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" event={"ID":"4b376785-4691-42e6-a35d-14bb5de40dd5","Type":"ContainerStarted","Data":"0d227546f57c6fa00891483f80e37fa0d8fc2ae35ce5a5a3dd68f75bda9c8e48"} Oct 31 01:43:57.678310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:57.677833 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" event={"ID":"4b376785-4691-42e6-a35d-14bb5de40dd5","Type":"ContainerStarted","Data":"e69f2aa64a97f70625458bbd53f8956d54f1c46d9311f8839fdd3e6d6beccfca"} Oct 31 01:43:57.793224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:57.793168 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" podStartSLOduration=1.793154893 podStartE2EDuration="1.793154893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:57.790921019 +0000 UTC m=+2809.597888562" watchObservedRunningTime="2026-10-31 01:43:57.793154893 +0000 UTC m=+2809.600122436" Oct 31 01:43:58.685046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:58.685009 5756 generic.go:356] "Generic (PLEG): container finished" podID="4b376785-4691-42e6-a35d-14bb5de40dd5" containerID="0d227546f57c6fa00891483f80e37fa0d8fc2ae35ce5a5a3dd68f75bda9c8e48" exitCode=0 Oct 31 01:43:58.685441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:58.685094 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" event={"ID":"4b376785-4691-42e6-a35d-14bb5de40dd5","Type":"ContainerDied","Data":"0d227546f57c6fa00891483f80e37fa0d8fc2ae35ce5a5a3dd68f75bda9c8e48"} Oct 31 01:43:59.836605 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.836581 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:43:59.963477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.963402 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca\") pod \"4b376785-4691-42e6-a35d-14bb5de40dd5\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " Oct 31 01:43:59.963477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.963462 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca\") pod \"4b376785-4691-42e6-a35d-14bb5de40dd5\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " Oct 31 01:43:59.963717 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.963569 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9\" (UniqueName: \"kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9\") pod \"4b376785-4691-42e6-a35d-14bb5de40dd5\" (UID: \"4b376785-4691-42e6-a35d-14bb5de40dd5\") " Oct 31 01:43:59.963816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.963793 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca" pod "4b376785-4691-42e6-a35d-14bb5de40dd5" (UID: "4b376785-4691-42e6-a35d-14bb5de40dd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:43:59.964390 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.964364 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:43:59.965775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.965749 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca" pod "4b376785-4691-42e6-a35d-14bb5de40dd5" (UID: "4b376785-4691-42e6-a35d-14bb5de40dd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:43:59.965962 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:43:59.965834 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9" pod "4b376785-4691-42e6-a35d-14bb5de40dd5" (UID: "4b376785-4691-42e6-a35d-14bb5de40dd5"). InnerVolumeSpecName "kube-api-access-qkwg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:00.065600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:00.065561 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b376785-4691-42e6-a35d-14bb5de40dd5-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:00.065600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:00.065593 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkwg9\" (UniqueName: \"kubernetes.io/projected/4b376785-4691-42e6-a35d-14bb5de40dd5-kube-api-access-qkwg9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:00.698761 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:00.698687 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" Oct 31 01:44:00.698761 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:00.698695 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8" event={"ID":"4b376785-4691-42e6-a35d-14bb5de40dd5","Type":"ContainerDied","Data":"e69f2aa64a97f70625458bbd53f8956d54f1c46d9311f8839fdd3e6d6beccfca"} Oct 31 01:44:00.698995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:00.698769 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69f2aa64a97f70625458bbd53f8956d54f1c46d9311f8839fdd3e6d6beccfca" Oct 31 01:44:02.066788 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:44:02.066758 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:44:05.603930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.603894 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp"] Oct 31 01:44:05.604421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.604406 5756 cpu_manager.go:414] "containerMap: removing container" podUID="4b376785-4691-42e6-a35d-14bb5de40dd5" containerName="revoke-keys" Oct 31 01:44:05.604466 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.604424 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b376785-4691-42e6-a35d-14bb5de40dd5" containerName="revoke-keys" Oct 31 01:44:05.604553 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.604543 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b376785-4691-42e6-a35d-14bb5de40dd5" containerName="revoke-keys" Oct 31 01:44:05.607589 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.607574 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.662435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.662380 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:44:05.665246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.665218 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp"] Oct 31 01:44:05.711802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.711775 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:44:05.730836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.730809 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.731033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.730946 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8wq\" (UniqueName: \"kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.731033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.730994 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.832164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.832127 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.832361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.832238 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.832361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.832324 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8wq\" (UniqueName: \"kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.832991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.832971 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.834762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.834741 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.890653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.890589 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8wq\" (UniqueName: \"kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq\") pod \"maas-api-revoke-sub-775be981-nzvsp\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:05.921675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:05.921642 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:06.295314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:06.295127 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp"] Oct 31 01:44:06.297918 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:06.297889 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8436aee_129f_4df4_86b4_9754971c7fd0.slice/crio-3d23f55dd19b4fa576b77126fb259fdfcd7d8e5b1d6c1ef9cc461e7bdb67a497 WatchSource:0}: Error finding container 3d23f55dd19b4fa576b77126fb259fdfcd7d8e5b1d6c1ef9cc461e7bdb67a497: Status 404 returned error can't find the container with id 3d23f55dd19b4fa576b77126fb259fdfcd7d8e5b1d6c1ef9cc461e7bdb67a497 Oct 31 01:44:06.739059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:06.739016 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" event={"ID":"f8436aee-129f-4df4-86b4-9754971c7fd0","Type":"ContainerStarted","Data":"d85709a42ea80dd479e4c778403f40535d1a45a5f9ea99fabbd2c602f3516937"} Oct 31 01:44:06.739530 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:06.739086 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" event={"ID":"f8436aee-129f-4df4-86b4-9754971c7fd0","Type":"ContainerStarted","Data":"3d23f55dd19b4fa576b77126fb259fdfcd7d8e5b1d6c1ef9cc461e7bdb67a497"} Oct 31 01:44:06.859818 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:06.859726 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" podStartSLOduration=1.8597128760000001 podStartE2EDuration="1.859712876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:06.858567717 +0000 UTC m=+2818.665535282" watchObservedRunningTime="2026-10-31 01:44:06.859712876 +0000 UTC m=+2818.666680420" Oct 31 01:44:07.746387 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:07.746312 5756 generic.go:356] "Generic (PLEG): container finished" podID="f8436aee-129f-4df4-86b4-9754971c7fd0" containerID="d85709a42ea80dd479e4c778403f40535d1a45a5f9ea99fabbd2c602f3516937" exitCode=0 Oct 31 01:44:07.746773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:07.746397 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" event={"ID":"f8436aee-129f-4df4-86b4-9754971c7fd0","Type":"ContainerDied","Data":"d85709a42ea80dd479e4c778403f40535d1a45a5f9ea99fabbd2c602f3516937"} Oct 31 01:44:08.904519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.904495 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:08.967670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.967640 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca\") pod \"f8436aee-129f-4df4-86b4-9754971c7fd0\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " Oct 31 01:44:08.967839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.967741 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca\") pod \"f8436aee-129f-4df4-86b4-9754971c7fd0\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " Oct 31 01:44:08.967839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.967832 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq\" (UniqueName: \"kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq\") pod \"f8436aee-129f-4df4-86b4-9754971c7fd0\" (UID: \"f8436aee-129f-4df4-86b4-9754971c7fd0\") " Oct 31 01:44:08.968102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.968075 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca" pod "f8436aee-129f-4df4-86b4-9754971c7fd0" (UID: "f8436aee-129f-4df4-86b4-9754971c7fd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:08.968236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.968221 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:08.970028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.970007 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca" pod "f8436aee-129f-4df4-86b4-9754971c7fd0" (UID: "f8436aee-129f-4df4-86b4-9754971c7fd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:08.970110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:08.970083 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq" pod "f8436aee-129f-4df4-86b4-9754971c7fd0" (UID: "f8436aee-129f-4df4-86b4-9754971c7fd0"). InnerVolumeSpecName "kube-api-access-ds8wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:09.069626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:09.069590 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8436aee-129f-4df4-86b4-9754971c7fd0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:09.069626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:09.069622 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds8wq\" (UniqueName: \"kubernetes.io/projected/f8436aee-129f-4df4-86b4-9754971c7fd0-kube-api-access-ds8wq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:09.759540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:09.759505 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" event={"ID":"f8436aee-129f-4df4-86b4-9754971c7fd0","Type":"ContainerDied","Data":"3d23f55dd19b4fa576b77126fb259fdfcd7d8e5b1d6c1ef9cc461e7bdb67a497"} Oct 31 01:44:09.759540 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:09.759529 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp" Oct 31 01:44:09.759748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:09.759578 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d23f55dd19b4fa576b77126fb259fdfcd7d8e5b1d6c1ef9cc461e7bdb67a497" Oct 31 01:44:10.072665 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:10.072629 5756 scope.go:122] "RemoveContainer" containerID="62ce6f80c9c3e3c63f8e36150cdc625057e1cafe8e6f9e5f7e2809f5b5a9aba2" Oct 31 01:44:12.968711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:12.968677 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj"] Oct 31 01:44:12.969247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:12.969232 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f8436aee-129f-4df4-86b4-9754971c7fd0" containerName="revoke-keys" Oct 31 01:44:12.969291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:12.969248 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8436aee-129f-4df4-86b4-9754971c7fd0" containerName="revoke-keys" Oct 31 01:44:12.969367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:12.969357 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8436aee-129f-4df4-86b4-9754971c7fd0" containerName="revoke-keys" Oct 31 01:44:12.974242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:12.974223 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.016286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.016258 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2zw\" (UniqueName: \"kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.016452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.016380 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.016452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.016436 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.033116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.033084 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj"] Oct 31 01:44:13.078497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.078470 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:44:13.078684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.078623 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:44:13.117237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.117206 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.117392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.117264 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.117392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.117334 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2zw\" (UniqueName: \"kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.117829 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.117810 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.119781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.119760 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.175696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.175667 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2zw\" (UniqueName: \"kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw\") pod \"maas-api-revoke-sub-76e42945-7f8mj\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.287396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.287375 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:13.664943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.664849 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj"] Oct 31 01:44:13.668322 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:13.668290 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1545929_62cc_4e1a_82b4_3371baf04b1d.slice/crio-332c11dbcd20c83e4a1b9dcab0070ecacdd0b96f7541d96ab0b7e384aef097d2 WatchSource:0}: Error finding container 332c11dbcd20c83e4a1b9dcab0070ecacdd0b96f7541d96ab0b7e384aef097d2: Status 404 returned error can't find the container with id 332c11dbcd20c83e4a1b9dcab0070ecacdd0b96f7541d96ab0b7e384aef097d2 Oct 31 01:44:13.786588 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:13.786555 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" event={"ID":"d1545929-62cc-4e1a-82b4-3371baf04b1d","Type":"ContainerStarted","Data":"332c11dbcd20c83e4a1b9dcab0070ecacdd0b96f7541d96ab0b7e384aef097d2"} Oct 31 01:44:14.205811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:14.205721 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:44:14.796722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:14.796686 5756 generic.go:356] "Generic (PLEG): container finished" podID="d1545929-62cc-4e1a-82b4-3371baf04b1d" containerID="588c36e6ac006aee83abbf47832050d4cbc298ef7426239682cfadc94fecff8d" exitCode=0 Oct 31 01:44:14.796923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:14.796747 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" event={"ID":"d1545929-62cc-4e1a-82b4-3371baf04b1d","Type":"ContainerDied","Data":"588c36e6ac006aee83abbf47832050d4cbc298ef7426239682cfadc94fecff8d"} Oct 31 01:44:15.990439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:15.990411 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:16.054467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.054376 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca\") pod \"d1545929-62cc-4e1a-82b4-3371baf04b1d\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " Oct 31 01:44:16.054784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.054761 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw\" (UniqueName: \"kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw\") pod \"d1545929-62cc-4e1a-82b4-3371baf04b1d\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " Oct 31 01:44:16.055068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.055050 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca\") pod \"d1545929-62cc-4e1a-82b4-3371baf04b1d\" (UID: \"d1545929-62cc-4e1a-82b4-3371baf04b1d\") " Oct 31 01:44:16.055667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.055531 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca" pod "d1545929-62cc-4e1a-82b4-3371baf04b1d" (UID: "d1545929-62cc-4e1a-82b4-3371baf04b1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:16.056274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.056251 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:16.057790 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.057763 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw" pod "d1545929-62cc-4e1a-82b4-3371baf04b1d" (UID: "d1545929-62cc-4e1a-82b4-3371baf04b1d"). InnerVolumeSpecName "kube-api-access-6m2zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:16.059824 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.059761 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca" pod "d1545929-62cc-4e1a-82b4-3371baf04b1d" (UID: "d1545929-62cc-4e1a-82b4-3371baf04b1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:16.157497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.157462 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2zw\" (UniqueName: \"kubernetes.io/projected/d1545929-62cc-4e1a-82b4-3371baf04b1d-kube-api-access-6m2zw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:16.157497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.157498 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1545929-62cc-4e1a-82b4-3371baf04b1d-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:16.413437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.413404 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr"] Oct 31 01:44:16.413960 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.413946 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d1545929-62cc-4e1a-82b4-3371baf04b1d" containerName="revoke-keys" Oct 31 01:44:16.414019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.413961 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1545929-62cc-4e1a-82b4-3371baf04b1d" containerName="revoke-keys" Oct 31 01:44:16.414059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.414046 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1545929-62cc-4e1a-82b4-3371baf04b1d" containerName="revoke-keys" Oct 31 01:44:16.417501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.417477 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.460652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.460615 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.460846 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.460678 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.460846 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.460820 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcf9x\" (UniqueName: \"kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.476322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.476290 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr"] Oct 31 01:44:16.561987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.561949 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.562155 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.562058 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.562233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.562167 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcf9x\" (UniqueName: \"kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.562696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.562666 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.564688 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.564667 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.619244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.619166 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcf9x\" (UniqueName: \"kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x\") pod \"maas-api-revoke-sub-3658b19a-6mcqr\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.730701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.730655 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:16.811586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.811535 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" event={"ID":"d1545929-62cc-4e1a-82b4-3371baf04b1d","Type":"ContainerDied","Data":"332c11dbcd20c83e4a1b9dcab0070ecacdd0b96f7541d96ab0b7e384aef097d2"} Oct 31 01:44:16.811745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.811609 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332c11dbcd20c83e4a1b9dcab0070ecacdd0b96f7541d96ab0b7e384aef097d2" Oct 31 01:44:16.811745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:16.811548 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj" Oct 31 01:44:17.114408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:17.114385 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr"] Oct 31 01:44:17.116514 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:17.116488 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166a6c2a_c77b_4c09_8a8e_600c47dc2ddc.slice/crio-986100e4e78a08b2ac61148b0a149f0583a9344d71e10e20c4c2f818ce48b345 WatchSource:0}: Error finding container 986100e4e78a08b2ac61148b0a149f0583a9344d71e10e20c4c2f818ce48b345: Status 404 returned error can't find the container with id 986100e4e78a08b2ac61148b0a149f0583a9344d71e10e20c4c2f818ce48b345 Oct 31 01:44:17.819782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:17.819735 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" event={"ID":"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc","Type":"ContainerStarted","Data":"4549eaa0f7ee99546f64108808d8db6ac2b16644754797b08e7c5d14f842d9cc"} Oct 31 01:44:17.819990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:17.819808 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" event={"ID":"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc","Type":"ContainerStarted","Data":"986100e4e78a08b2ac61148b0a149f0583a9344d71e10e20c4c2f818ce48b345"} Oct 31 01:44:17.934339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:17.934289 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" podStartSLOduration=1.9342731 podStartE2EDuration="1.9342731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:17.931931138 +0000 UTC m=+2829.738898681" watchObservedRunningTime="2026-10-31 01:44:17.9342731 +0000 UTC m=+2829.741240643" Oct 31 01:44:18.827570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:18.827532 5756 generic.go:356] "Generic (PLEG): container finished" podID="166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" containerID="4549eaa0f7ee99546f64108808d8db6ac2b16644754797b08e7c5d14f842d9cc" exitCode=0 Oct 31 01:44:18.828032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:18.827596 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" event={"ID":"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc","Type":"ContainerDied","Data":"4549eaa0f7ee99546f64108808d8db6ac2b16644754797b08e7c5d14f842d9cc"} Oct 31 01:44:19.973338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:19.973315 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:20.101498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.101417 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca\") pod \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " Oct 31 01:44:20.101672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.101522 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca\") pod \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " Oct 31 01:44:20.101672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.101632 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x\" (UniqueName: \"kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x\") pod \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\" (UID: \"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc\") " Oct 31 01:44:20.101855 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.101829 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca" pod "166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" (UID: "166a6c2a-c77b-4c09-8a8e-600c47dc2ddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:20.103694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.103661 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca" pod "166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" (UID: "166a6c2a-c77b-4c09-8a8e-600c47dc2ddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:20.103819 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.103792 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x" pod "166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" (UID: "166a6c2a-c77b-4c09-8a8e-600c47dc2ddc"). InnerVolumeSpecName "kube-api-access-vcf9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:20.202770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.202738 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:20.202770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.202769 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcf9x\" (UniqueName: \"kubernetes.io/projected/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-kube-api-access-vcf9x\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:20.202770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.202780 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:20.841986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.841955 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" Oct 31 01:44:20.842144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.841988 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr" event={"ID":"166a6c2a-c77b-4c09-8a8e-600c47dc2ddc","Type":"ContainerDied","Data":"986100e4e78a08b2ac61148b0a149f0583a9344d71e10e20c4c2f818ce48b345"} Oct 31 01:44:20.842144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:20.842031 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="986100e4e78a08b2ac61148b0a149f0583a9344d71e10e20c4c2f818ce48b345" Oct 31 01:44:23.727666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.727637 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz"] Oct 31 01:44:23.728395 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.728373 5756 cpu_manager.go:414] "containerMap: removing container" podUID="166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" containerName="revoke-keys" Oct 31 01:44:23.728495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.728397 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" containerName="revoke-keys" Oct 31 01:44:23.728551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.728516 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" containerName="revoke-keys" Oct 31 01:44:23.732325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.732307 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.784313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.784254 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:44:23.833676 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.833640 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:44:23.836229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.836207 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz"] Oct 31 01:44:23.842183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.842159 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.842263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.842234 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.842379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.842360 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.943072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.943037 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.943261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.943145 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.943261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.943183 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.943705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.943684 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:23.945658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:23.945635 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:24.000458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:24.000392 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d\") pod \"maas-api-revoke-sub-317e77d9-mftzz\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:24.046038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:24.046010 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:24.419495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:24.419470 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz"] Oct 31 01:44:24.421661 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:24.421627 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48594c70_93a6_4a03_99b1_c58fa948883f.slice/crio-0145d24deeea6cc53b7bf20e24c857384808c8cd89c6a44a4f17d020c8e02d4e WatchSource:0}: Error finding container 0145d24deeea6cc53b7bf20e24c857384808c8cd89c6a44a4f17d020c8e02d4e: Status 404 returned error can't find the container with id 0145d24deeea6cc53b7bf20e24c857384808c8cd89c6a44a4f17d020c8e02d4e Oct 31 01:44:24.868084 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:24.868051 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" event={"ID":"48594c70-93a6-4a03-99b1-c58fa948883f","Type":"ContainerStarted","Data":"9b828371b7238baa820ddeb3e97fd5544120453ce4805a7ab87a2b0fcb546bca"} Oct 31 01:44:24.868565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:24.868105 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" event={"ID":"48594c70-93a6-4a03-99b1-c58fa948883f","Type":"ContainerStarted","Data":"0145d24deeea6cc53b7bf20e24c857384808c8cd89c6a44a4f17d020c8e02d4e"} Oct 31 01:44:24.984172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:24.984122 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" podStartSLOduration=1.984108858 podStartE2EDuration="1.984108858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:24.983639833 +0000 UTC m=+2836.790607445" watchObservedRunningTime="2026-10-31 01:44:24.984108858 +0000 UTC m=+2836.791076401" Oct 31 01:44:25.875574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:25.875536 5756 generic.go:356] "Generic (PLEG): container finished" podID="48594c70-93a6-4a03-99b1-c58fa948883f" containerID="9b828371b7238baa820ddeb3e97fd5544120453ce4805a7ab87a2b0fcb546bca" exitCode=0 Oct 31 01:44:25.875995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:25.875620 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" event={"ID":"48594c70-93a6-4a03-99b1-c58fa948883f","Type":"ContainerDied","Data":"9b828371b7238baa820ddeb3e97fd5544120453ce4805a7ab87a2b0fcb546bca"} Oct 31 01:44:27.034181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.034154 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:27.179524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.179440 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca\") pod \"48594c70-93a6-4a03-99b1-c58fa948883f\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " Oct 31 01:44:27.179524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.179520 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d\") pod \"48594c70-93a6-4a03-99b1-c58fa948883f\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " Oct 31 01:44:27.179777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.179630 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca\") pod \"48594c70-93a6-4a03-99b1-c58fa948883f\" (UID: \"48594c70-93a6-4a03-99b1-c58fa948883f\") " Oct 31 01:44:27.180027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.180002 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca" pod "48594c70-93a6-4a03-99b1-c58fa948883f" (UID: "48594c70-93a6-4a03-99b1-c58fa948883f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:27.181837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.181814 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d" pod "48594c70-93a6-4a03-99b1-c58fa948883f" (UID: "48594c70-93a6-4a03-99b1-c58fa948883f"). InnerVolumeSpecName "kube-api-access-drx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:27.181986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.181894 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca" pod "48594c70-93a6-4a03-99b1-c58fa948883f" (UID: "48594c70-93a6-4a03-99b1-c58fa948883f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:27.281229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.281198 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:27.281229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.281225 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drx7d\" (UniqueName: \"kubernetes.io/projected/48594c70-93a6-4a03-99b1-c58fa948883f-kube-api-access-drx7d\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:27.281229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.281236 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48594c70-93a6-4a03-99b1-c58fa948883f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:27.889691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.889658 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" Oct 31 01:44:27.889691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.889665 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz" event={"ID":"48594c70-93a6-4a03-99b1-c58fa948883f","Type":"ContainerDied","Data":"0145d24deeea6cc53b7bf20e24c857384808c8cd89c6a44a4f17d020c8e02d4e"} Oct 31 01:44:27.889938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:27.889708 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0145d24deeea6cc53b7bf20e24c857384808c8cd89c6a44a4f17d020c8e02d4e" Oct 31 01:44:32.066589 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:44:32.066553 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:44:32.690680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.690643 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc"] Oct 31 01:44:32.691193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.691178 5756 cpu_manager.go:414] "containerMap: removing container" podUID="48594c70-93a6-4a03-99b1-c58fa948883f" containerName="revoke-keys" Oct 31 01:44:32.691238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.691196 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48594c70-93a6-4a03-99b1-c58fa948883f" containerName="revoke-keys" Oct 31 01:44:32.691303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.691293 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="48594c70-93a6-4a03-99b1-c58fa948883f" containerName="revoke-keys" Oct 31 01:44:32.694786 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.694769 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.747123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.747094 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:44:32.747271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.747202 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:44:32.800600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.800559 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc"] Oct 31 01:44:32.846725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.846696 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.846929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.846845 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.846929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.846918 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5ks\" (UniqueName: \"kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.947986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.947898 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.947986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.947970 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5ks\" (UniqueName: \"kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.948234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.948051 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.948557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.948531 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:32.950514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:32.950484 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:33.005127 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:33.005093 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5ks\" (UniqueName: \"kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks\") pod \"maas-api-revoke-sub-767c655c-dx4xc\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:33.007790 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:33.007772 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:33.387435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:33.387409 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc"] Oct 31 01:44:33.389710 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:33.389683 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda96ade5d_922b_4e3a_9ae7_d435ca0e0a72.slice/crio-b73a90f3bd7084a8d8cebe3311368c6a17a7e72bed89e352c5bc19004a2561f9 WatchSource:0}: Error finding container b73a90f3bd7084a8d8cebe3311368c6a17a7e72bed89e352c5bc19004a2561f9: Status 404 returned error can't find the container with id b73a90f3bd7084a8d8cebe3311368c6a17a7e72bed89e352c5bc19004a2561f9 Oct 31 01:44:33.941021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:33.940921 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" event={"ID":"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72","Type":"ContainerStarted","Data":"507a2a0a27ea154e49297385c348817977dccef85d1e3524c3d417ca9a09a967"} Oct 31 01:44:33.941021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:33.941017 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" event={"ID":"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72","Type":"ContainerStarted","Data":"b73a90f3bd7084a8d8cebe3311368c6a17a7e72bed89e352c5bc19004a2561f9"} Oct 31 01:44:34.064580 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:34.064520 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" podStartSLOduration=2.064500459 podStartE2EDuration="2.064500459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:34.062591109 +0000 UTC m=+2845.869558655" watchObservedRunningTime="2026-10-31 01:44:34.064500459 +0000 UTC m=+2845.871468003" Oct 31 01:44:34.948452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:34.948416 5756 generic.go:356] "Generic (PLEG): container finished" podID="a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" containerID="507a2a0a27ea154e49297385c348817977dccef85d1e3524c3d417ca9a09a967" exitCode=0 Oct 31 01:44:34.948892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:34.948507 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" event={"ID":"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72","Type":"ContainerDied","Data":"507a2a0a27ea154e49297385c348817977dccef85d1e3524c3d417ca9a09a967"} Oct 31 01:44:36.094247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.094221 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:36.185302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.185267 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca\") pod \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " Oct 31 01:44:36.185302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.185310 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca\") pod \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " Oct 31 01:44:36.185534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.185375 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks\" (UniqueName: \"kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks\") pod \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\" (UID: \"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72\") " Oct 31 01:44:36.185723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.185699 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca" pod "a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" (UID: "a96ade5d-922b-4e3a-9ae7-d435ca0e0a72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:36.185902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.185856 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:36.187668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.187640 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks" pod "a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" (UID: "a96ade5d-922b-4e3a-9ae7-d435ca0e0a72"). InnerVolumeSpecName "kube-api-access-lg5ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:36.187783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.187705 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca" pod "a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" (UID: "a96ade5d-922b-4e3a-9ae7-d435ca0e0a72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:36.286502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.286477 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5ks\" (UniqueName: \"kubernetes.io/projected/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-kube-api-access-lg5ks\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:36.286657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.286504 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:36.961900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.961847 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" Oct 31 01:44:36.961900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.961889 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc" event={"ID":"a96ade5d-922b-4e3a-9ae7-d435ca0e0a72","Type":"ContainerDied","Data":"b73a90f3bd7084a8d8cebe3311368c6a17a7e72bed89e352c5bc19004a2561f9"} Oct 31 01:44:36.962119 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:36.961934 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73a90f3bd7084a8d8cebe3311368c6a17a7e72bed89e352c5bc19004a2561f9" Oct 31 01:44:40.179539 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.179506 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9"] Oct 31 01:44:40.180093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.180079 5756 cpu_manager.go:414] "containerMap: removing container" podUID="a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" containerName="revoke-keys" Oct 31 01:44:40.180138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.180095 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" containerName="revoke-keys" Oct 31 01:44:40.180196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.180184 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" containerName="revoke-keys" Oct 31 01:44:40.183775 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.183748 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.228246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.228218 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt49c\" (UniqueName: \"kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.228409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.228254 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.228409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.228361 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.239397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.239375 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:44:40.294718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.294682 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:44:40.297338 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.297309 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9"] Oct 31 01:44:40.329325 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.329291 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt49c\" (UniqueName: \"kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.329508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.329346 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.329508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.329404 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.330165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.330139 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.331941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.331920 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.385839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.385812 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt49c\" (UniqueName: \"kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c\") pod \"maas-api-revoke-sub-70fa5b34-27tc9\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.497117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.497040 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:40.874168 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:40.874136 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c118b5_48ff_432a_b8a9_dc52e06979f4.slice/crio-b9d21b85578e68bbd97eb87b645012d1c1ee9fbd3bb8444c3feba9265b9e6c94 WatchSource:0}: Error finding container b9d21b85578e68bbd97eb87b645012d1c1ee9fbd3bb8444c3feba9265b9e6c94: Status 404 returned error can't find the container with id b9d21b85578e68bbd97eb87b645012d1c1ee9fbd3bb8444c3feba9265b9e6c94 Oct 31 01:44:40.875973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.875946 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9"] Oct 31 01:44:40.987922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:40.987852 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" event={"ID":"24c118b5-48ff-432a-b8a9-dc52e06979f4","Type":"ContainerStarted","Data":"b9d21b85578e68bbd97eb87b645012d1c1ee9fbd3bb8444c3feba9265b9e6c94"} Oct 31 01:44:41.998401 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:41.998361 5756 generic.go:356] "Generic (PLEG): container finished" podID="24c118b5-48ff-432a-b8a9-dc52e06979f4" containerID="4395fb3ff514c97d85633bc19b78855f243150e50efef0ba2efe5fd69b123d78" exitCode=0 Oct 31 01:44:41.998789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:41.998587 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" event={"ID":"24c118b5-48ff-432a-b8a9-dc52e06979f4","Type":"ContainerDied","Data":"4395fb3ff514c97d85633bc19b78855f243150e50efef0ba2efe5fd69b123d78"} Oct 31 01:44:43.151318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.151296 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:43.267745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.267658 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca\") pod \"24c118b5-48ff-432a-b8a9-dc52e06979f4\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " Oct 31 01:44:43.267745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.267740 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca\") pod \"24c118b5-48ff-432a-b8a9-dc52e06979f4\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " Oct 31 01:44:43.268010 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.267792 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c\" (UniqueName: \"kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c\") pod \"24c118b5-48ff-432a-b8a9-dc52e06979f4\" (UID: \"24c118b5-48ff-432a-b8a9-dc52e06979f4\") " Oct 31 01:44:43.268178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.268148 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca" pod "24c118b5-48ff-432a-b8a9-dc52e06979f4" (UID: "24c118b5-48ff-432a-b8a9-dc52e06979f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:43.270009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.269980 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca" pod "24c118b5-48ff-432a-b8a9-dc52e06979f4" (UID: "24c118b5-48ff-432a-b8a9-dc52e06979f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:43.270122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.270106 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c" pod "24c118b5-48ff-432a-b8a9-dc52e06979f4" (UID: "24c118b5-48ff-432a-b8a9-dc52e06979f4"). InnerVolumeSpecName "kube-api-access-pt49c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:43.369358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.369323 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt49c\" (UniqueName: \"kubernetes.io/projected/24c118b5-48ff-432a-b8a9-dc52e06979f4-kube-api-access-pt49c\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:43.369358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.369351 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:43.369358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:43.369361 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c118b5-48ff-432a-b8a9-dc52e06979f4-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:44.012643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:44.012614 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" Oct 31 01:44:44.012843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:44.012641 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9" event={"ID":"24c118b5-48ff-432a-b8a9-dc52e06979f4","Type":"ContainerDied","Data":"b9d21b85578e68bbd97eb87b645012d1c1ee9fbd3bb8444c3feba9265b9e6c94"} Oct 31 01:44:44.012843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:44.012686 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9d21b85578e68bbd97eb87b645012d1c1ee9fbd3bb8444c3feba9265b9e6c94" Oct 31 01:44:54.905351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.905317 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm"] Oct 31 01:44:54.905922 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.905906 5756 cpu_manager.go:414] "containerMap: removing container" podUID="24c118b5-48ff-432a-b8a9-dc52e06979f4" containerName="revoke-keys" Oct 31 01:44:54.905976 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.905925 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c118b5-48ff-432a-b8a9-dc52e06979f4" containerName="revoke-keys" Oct 31 01:44:54.906037 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.906026 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c118b5-48ff-432a-b8a9-dc52e06979f4" containerName="revoke-keys" Oct 31 01:44:54.909652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.909637 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:54.966116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.966081 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm"] Oct 31 01:44:54.998940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.998904 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:54.999096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.999034 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8xs\" (UniqueName: \"kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:54.999096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:54.999063 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.011946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.011923 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:44:55.012043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.011997 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:44:55.100500 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.100467 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.100660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.100545 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8xs\" (UniqueName: \"kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.100660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.100569 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.101111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.101088 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.102894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.102853 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.157527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.157463 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8xs\" (UniqueName: \"kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs\") pod \"maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.223568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.223541 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:55.600080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:55.600053 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm"] Oct 31 01:44:55.601541 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:44:55.601517 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6185b5_5698_4d84_9999_b97ddc33873f.slice/crio-20f48d275150a0df10417ba2c96294b77eb15afd80282f5c627d8e1a7d7730a4 WatchSource:0}: Error finding container 20f48d275150a0df10417ba2c96294b77eb15afd80282f5c627d8e1a7d7730a4: Status 404 returned error can't find the container with id 20f48d275150a0df10417ba2c96294b77eb15afd80282f5c627d8e1a7d7730a4 Oct 31 01:44:56.091927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:56.091886 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" event={"ID":"cc6185b5-5698-4d84-9999-b97ddc33873f","Type":"ContainerStarted","Data":"58df49b8cd5a004de41c52eaa8feb4042c1a3cfcc358f20ce769c953bef9885a"} Oct 31 01:44:56.092407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:56.091954 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" event={"ID":"cc6185b5-5698-4d84-9999-b97ddc33873f","Type":"ContainerStarted","Data":"20f48d275150a0df10417ba2c96294b77eb15afd80282f5c627d8e1a7d7730a4"} Oct 31 01:44:56.207007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:56.206897 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" podStartSLOduration=2.206882398 podStartE2EDuration="2.206882398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:56.205144276 +0000 UTC m=+2868.012111818" watchObservedRunningTime="2026-10-31 01:44:56.206882398 +0000 UTC m=+2868.013849941" Oct 31 01:44:57.099732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:57.099700 5756 generic.go:356] "Generic (PLEG): container finished" podID="cc6185b5-5698-4d84-9999-b97ddc33873f" containerID="58df49b8cd5a004de41c52eaa8feb4042c1a3cfcc358f20ce769c953bef9885a" exitCode=0 Oct 31 01:44:57.100174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:57.099793 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" event={"ID":"cc6185b5-5698-4d84-9999-b97ddc33873f","Type":"ContainerDied","Data":"58df49b8cd5a004de41c52eaa8feb4042c1a3cfcc358f20ce769c953bef9885a"} Oct 31 01:44:58.249257 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.249234 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:58.337102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.337074 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs\" (UniqueName: \"kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs\") pod \"cc6185b5-5698-4d84-9999-b97ddc33873f\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " Oct 31 01:44:58.337243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.337122 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca\") pod \"cc6185b5-5698-4d84-9999-b97ddc33873f\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " Oct 31 01:44:58.337243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.337152 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca\") pod \"cc6185b5-5698-4d84-9999-b97ddc33873f\" (UID: \"cc6185b5-5698-4d84-9999-b97ddc33873f\") " Oct 31 01:44:58.337503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.337473 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca" pod "cc6185b5-5698-4d84-9999-b97ddc33873f" (UID: "cc6185b5-5698-4d84-9999-b97ddc33873f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:44:58.339353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.339326 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs" pod "cc6185b5-5698-4d84-9999-b97ddc33873f" (UID: "cc6185b5-5698-4d84-9999-b97ddc33873f"). InnerVolumeSpecName "kube-api-access-kr8xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:44:58.339353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.339334 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca" pod "cc6185b5-5698-4d84-9999-b97ddc33873f" (UID: "cc6185b5-5698-4d84-9999-b97ddc33873f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:44:58.439056 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.438968 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:58.439056 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.439001 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8xs\" (UniqueName: \"kubernetes.io/projected/cc6185b5-5698-4d84-9999-b97ddc33873f-kube-api-access-kr8xs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:58.439056 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:58.439016 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6185b5-5698-4d84-9999-b97ddc33873f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:44:59.114718 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:59.114692 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" Oct 31 01:44:59.114951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:59.114690 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm" event={"ID":"cc6185b5-5698-4d84-9999-b97ddc33873f","Type":"ContainerDied","Data":"20f48d275150a0df10417ba2c96294b77eb15afd80282f5c627d8e1a7d7730a4"} Oct 31 01:44:59.114951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:59.114825 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20f48d275150a0df10417ba2c96294b77eb15afd80282f5c627d8e1a7d7730a4" Oct 31 01:44:59.299190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:44:59.299151 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:45:00.179572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.179539 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw"] Oct 31 01:45:00.180795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.180768 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cc6185b5-5698-4d84-9999-b97ddc33873f" containerName="revoke-keys" Oct 31 01:45:00.181004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.180987 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6185b5-5698-4d84-9999-b97ddc33873f" containerName="revoke-keys" Oct 31 01:45:00.181284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.181267 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6185b5-5698-4d84-9999-b97ddc33873f" containerName="revoke-keys" Oct 31 01:45:00.186751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.186726 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.196593 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.196541 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5"] Oct 31 01:45:00.201298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.201274 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb"] Oct 31 01:45:00.201446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.201394 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.205600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.205576 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h"] Oct 31 01:45:00.205724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.205705 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.209348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.209313 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t"] Oct 31 01:45:00.210228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.209433 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.215011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.214990 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.227286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.227263 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4"] Oct 31 01:45:00.249813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.249261 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:45:00.270107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.270074 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp"] Oct 31 01:45:00.273456 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.273330 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.298763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.298678 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt"] Oct 31 01:45:00.299229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.299048 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.304422 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.304398 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.316356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.316329 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw"] Oct 31 01:45:00.316507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.316379 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5"] Oct 31 01:45:00.322142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.322113 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h"] Oct 31 01:45:00.328337 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.328310 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t"] Oct 31 01:45:00.341406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.341288 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb"] Oct 31 01:45:00.341719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.341699 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp"] Oct 31 01:45:00.377847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.377816 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.378166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.378138 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.378357 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.378337 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.378526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.378506 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6grn\" (UniqueName: \"kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.378681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.378659 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.378835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.378817 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbm9\" (UniqueName: \"kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.379008 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.378990 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.379175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.379156 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg4jm\" (UniqueName: \"kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm\") pod \"maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.379317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.379301 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvc9\" (UniqueName: \"kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9\") pod \"maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.379455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.379439 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrc4z\" (UniqueName: \"kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z\") pod \"maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.379647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.379628 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmxl\" (UniqueName: \"kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.379798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.379781 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.420346 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.420262 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4"] Oct 31 01:45:00.423130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.423106 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt"] Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487554 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6grn\" (UniqueName: \"kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487611 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487670 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbm9\" (UniqueName: \"kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487704 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8llw\" (UniqueName: \"kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487737 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487769 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg4jm\" (UniqueName: \"kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm\") pod \"maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487794 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvc9\" (UniqueName: \"kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9\") pod \"maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487822 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.487927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.487883 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrc4z\" (UniqueName: \"kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z\") pod \"maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488008 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmxl\" (UniqueName: \"kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488053 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488130 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488161 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmdkw\" (UniqueName: \"kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw\") pod \"maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488234 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488335 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.488674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.488376 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.490017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.489990 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.491485 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.491459 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.492945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.492923 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.495759 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.495725 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.496583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.495781 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.496732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.496232 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.566953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.566840 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg4jm\" (UniqueName: \"kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm\") pod \"maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.567807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.567762 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvc9\" (UniqueName: \"kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9\") pod \"maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.568602 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.568551 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbm9\" (UniqueName: \"kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9\") pod \"maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.571427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.571379 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrc4z\" (UniqueName: \"kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z\") pod \"maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.572445 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.572400 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmxl\" (UniqueName: \"kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl\") pod \"maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.574227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.574030 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:00.576234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.576191 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6grn\" (UniqueName: \"kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn\") pod \"maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.590224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.589822 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.590224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.589929 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmdkw\" (UniqueName: \"kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw\") pod \"maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.590224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.590086 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8llw\" (UniqueName: \"kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.590224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.590134 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.592525 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.591379 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.592525 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.592031 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.607479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.607456 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:00.619147 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.618938 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:00.678427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.678391 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8llw\" (UniqueName: \"kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw\") pod \"maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.679088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.679059 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmdkw\" (UniqueName: \"kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw\") pod \"maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.692919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.685908 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:00.805741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.805483 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:00.818176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.817937 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:00.827499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.827476 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:00.961767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.961531 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:00.972033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:00.972001 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h"] Oct 31 01:45:00.974775 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:00.974735 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c558a3_547a_4d2d_ac43_a19be6e4493d.slice/crio-496d0f08c91184586e95912832d64af5c39583b08215cf8efad59abee6fd2e0f WatchSource:0}: Error finding container 496d0f08c91184586e95912832d64af5c39583b08215cf8efad59abee6fd2e0f: Status 404 returned error can't find the container with id 496d0f08c91184586e95912832d64af5c39583b08215cf8efad59abee6fd2e0f Oct 31 01:45:01.037639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.037610 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t"] Oct 31 01:45:01.040368 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:01.040340 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c6093f_61cc_4b3c_97df_776555390fd7.slice/crio-ec1723c21cbddc295c389f422d9286d3a8c4af7ebbec95ff7d15007c2ff4c629 WatchSource:0}: Error finding container ec1723c21cbddc295c389f422d9286d3a8c4af7ebbec95ff7d15007c2ff4c629: Status 404 returned error can't find the container with id ec1723c21cbddc295c389f422d9286d3a8c4af7ebbec95ff7d15007c2ff4c629 Oct 31 01:45:01.104189 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.104150 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4"] Oct 31 01:45:01.130616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.130524 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" event={"ID":"77c6093f-61cc-4b3c-97df-776555390fd7","Type":"ContainerStarted","Data":"ec1723c21cbddc295c389f422d9286d3a8c4af7ebbec95ff7d15007c2ff4c629"} Oct 31 01:45:01.133182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.133124 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" event={"ID":"19c558a3-547a-4d2d-ac43-a19be6e4493d","Type":"ContainerStarted","Data":"496d0f08c91184586e95912832d64af5c39583b08215cf8efad59abee6fd2e0f"} Oct 31 01:45:01.135274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.134943 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" event={"ID":"377cfa2f-5923-4732-a87e-bedcc17090ba","Type":"ContainerStarted","Data":"2e784bd737fa4e5647ed4f3b9ac8eb01ff96601b8b7068c21990e036dc927142"} Oct 31 01:45:01.145770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.145750 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt"] Oct 31 01:45:01.231762 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:01.231715 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb53321bd_0aff_4ef5_8073_3a767cbd0329.slice/crio-7bd64314db922788ed42685d683aae337cd378dae70695e2adc1d4d0c06d8454 WatchSource:0}: Error finding container 7bd64314db922788ed42685d683aae337cd378dae70695e2adc1d4d0c06d8454: Status 404 returned error can't find the container with id 7bd64314db922788ed42685d683aae337cd378dae70695e2adc1d4d0c06d8454 Oct 31 01:45:01.244577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.244556 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw"] Oct 31 01:45:01.269740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.269717 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb"] Oct 31 01:45:01.276144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.276123 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5"] Oct 31 01:45:01.331978 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:01.331849 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd19de2_2772_40fb_ab58_2e35f5ef3c2a.slice/crio-3a6f12cc57b7dacf5241de2c2271765c1c394d50bc3bb604e242320c1afd268f WatchSource:0}: Error finding container 3a6f12cc57b7dacf5241de2c2271765c1c394d50bc3bb604e242320c1afd268f: Status 404 returned error can't find the container with id 3a6f12cc57b7dacf5241de2c2271765c1c394d50bc3bb604e242320c1afd268f Oct 31 01:45:01.365219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:01.365193 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp"] Oct 31 01:45:01.437624 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:01.437312 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7060ce_e966_4579_aa1d_964bef4a3efe.slice/crio-3f885c16ce71fb78a230cfbbaab289248189d4fe0c9af479f97cedad68d8f5bc WatchSource:0}: Error finding container 3f885c16ce71fb78a230cfbbaab289248189d4fe0c9af479f97cedad68d8f5bc: Status 404 returned error can't find the container with id 3f885c16ce71fb78a230cfbbaab289248189d4fe0c9af479f97cedad68d8f5bc Oct 31 01:45:02.130452 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:02.130404 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:45:02.164344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.164303 5756 generic.go:356] "Generic (PLEG): container finished" podID="77c6093f-61cc-4b3c-97df-776555390fd7" containerID="ec1f424dacca1882ad7aa0bc681ba7554ae999e43f6edbecbe95d4a4b2dc8525" exitCode=60 Oct 31 01:45:02.164455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.164367 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" event={"ID":"77c6093f-61cc-4b3c-97df-776555390fd7","Type":"ContainerDied","Data":"ec1f424dacca1882ad7aa0bc681ba7554ae999e43f6edbecbe95d4a4b2dc8525"} Oct 31 01:45:02.164837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.164714 5756 scope.go:122] "RemoveContainer" containerID="ec1f424dacca1882ad7aa0bc681ba7554ae999e43f6edbecbe95d4a4b2dc8525" Oct 31 01:45:02.166389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.166358 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" event={"ID":"9af9a4ba-e1ee-4481-89de-acb671353972","Type":"ContainerStarted","Data":"0c34c2d36c35cea3b7039cb4a6f90afcd6bb4110130974e9c974f1a46a660809"} Oct 31 01:45:02.168256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.168230 5756 generic.go:356] "Generic (PLEG): container finished" podID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerID="dd1212e23b30de5df75c83fb8675a60a982d453ea12b7aeeb10be644fd774ece" exitCode=60 Oct 31 01:45:02.168355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.168315 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" event={"ID":"19c558a3-547a-4d2d-ac43-a19be6e4493d","Type":"ContainerDied","Data":"dd1212e23b30de5df75c83fb8675a60a982d453ea12b7aeeb10be644fd774ece"} Oct 31 01:45:02.168559 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.168543 5756 scope.go:122] "RemoveContainer" containerID="dd1212e23b30de5df75c83fb8675a60a982d453ea12b7aeeb10be644fd774ece" Oct 31 01:45:02.171331 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.171198 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerStarted","Data":"3f885c16ce71fb78a230cfbbaab289248189d4fe0c9af479f97cedad68d8f5bc"} Oct 31 01:45:02.173102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.173076 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" event={"ID":"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a","Type":"ContainerStarted","Data":"3a6f12cc57b7dacf5241de2c2271765c1c394d50bc3bb604e242320c1afd268f"} Oct 31 01:45:02.174698 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.174664 5756 generic.go:356] "Generic (PLEG): container finished" podID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerID="2ac686646b339c0c6817f0283f2cbdfbdaabac0bf81d58a7773c4fe59d9a535a" exitCode=60 Oct 31 01:45:02.174794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.174713 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" event={"ID":"377cfa2f-5923-4732-a87e-bedcc17090ba","Type":"ContainerDied","Data":"2ac686646b339c0c6817f0283f2cbdfbdaabac0bf81d58a7773c4fe59d9a535a"} Oct 31 01:45:02.175043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.175024 5756 scope.go:122] "RemoveContainer" containerID="2ac686646b339c0c6817f0283f2cbdfbdaabac0bf81d58a7773c4fe59d9a535a" Oct 31 01:45:02.176601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.176576 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" event={"ID":"b53321bd-0aff-4ef5-8073-3a767cbd0329","Type":"ContainerStarted","Data":"7bd64314db922788ed42685d683aae337cd378dae70695e2adc1d4d0c06d8454"} Oct 31 01:45:02.178450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.178427 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" event={"ID":"b544add2-4eaa-4fc5-aef6-80a1dd872786","Type":"ContainerStarted","Data":"8b390324424496ae4ac58f0795e868a1e38c2f40087e39c28813cc9bfb75c80e"} Oct 31 01:45:02.501335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.501273 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" podStartSLOduration=2.501259663 podStartE2EDuration="2.501259663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.498542955 +0000 UTC m=+2874.305510499" watchObservedRunningTime="2026-10-31 01:45:02.501259663 +0000 UTC m=+2874.308227205" Oct 31 01:45:02.612609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:02.612498 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" podStartSLOduration=2.6124830340000003 podStartE2EDuration="2.612483034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.609351811 +0000 UTC m=+2874.416319365" watchObservedRunningTime="2026-10-31 01:45:02.612483034 +0000 UTC m=+2874.419450636" Oct 31 01:45:03.188925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.188828 5756 generic.go:356] "Generic (PLEG): container finished" podID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerID="6fd04c53c9ecc6eb6ca2297207e96f716fb3de0ecdbc96854eaebae345ffd624" exitCode=60 Oct 31 01:45:03.189072 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.188917 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" event={"ID":"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a","Type":"ContainerDied","Data":"6fd04c53c9ecc6eb6ca2297207e96f716fb3de0ecdbc96854eaebae345ffd624"} Oct 31 01:45:03.189259 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.189246 5756 scope.go:122] "RemoveContainer" containerID="6fd04c53c9ecc6eb6ca2297207e96f716fb3de0ecdbc96854eaebae345ffd624" Oct 31 01:45:03.191106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.191080 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" event={"ID":"377cfa2f-5923-4732-a87e-bedcc17090ba","Type":"ContainerStarted","Data":"18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2"} Oct 31 01:45:03.192853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.192829 5756 generic.go:356] "Generic (PLEG): container finished" podID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerID="95a1f18b81a621619287199d88106e60c239da6e6d913e6ae811901ae3ec7aa4" exitCode=60 Oct 31 01:45:03.192987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.192874 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" event={"ID":"b53321bd-0aff-4ef5-8073-3a767cbd0329","Type":"ContainerDied","Data":"95a1f18b81a621619287199d88106e60c239da6e6d913e6ae811901ae3ec7aa4"} Oct 31 01:45:03.193179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.193163 5756 scope.go:122] "RemoveContainer" containerID="95a1f18b81a621619287199d88106e60c239da6e6d913e6ae811901ae3ec7aa4" Oct 31 01:45:03.194574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.194552 5756 generic.go:356] "Generic (PLEG): container finished" podID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerID="f27bfcb8a190968e787900be7f346e4ad0abb9c84ffe847969581feb354233d6" exitCode=60 Oct 31 01:45:03.194709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.194688 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" event={"ID":"b544add2-4eaa-4fc5-aef6-80a1dd872786","Type":"ContainerDied","Data":"f27bfcb8a190968e787900be7f346e4ad0abb9c84ffe847969581feb354233d6"} Oct 31 01:45:03.194899 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.194885 5756 scope.go:122] "RemoveContainer" containerID="f27bfcb8a190968e787900be7f346e4ad0abb9c84ffe847969581feb354233d6" Oct 31 01:45:03.196667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.196648 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" event={"ID":"77c6093f-61cc-4b3c-97df-776555390fd7","Type":"ContainerStarted","Data":"546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e"} Oct 31 01:45:03.198349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.198328 5756 generic.go:356] "Generic (PLEG): container finished" podID="9af9a4ba-e1ee-4481-89de-acb671353972" containerID="cb0fb5bdff72faaddb4e50372de3849d17ed28ffe5d2c3dc0b1978a94b811869" exitCode=60 Oct 31 01:45:03.198458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.198369 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" event={"ID":"9af9a4ba-e1ee-4481-89de-acb671353972","Type":"ContainerDied","Data":"cb0fb5bdff72faaddb4e50372de3849d17ed28ffe5d2c3dc0b1978a94b811869"} Oct 31 01:45:03.198625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.198609 5756 scope.go:122] "RemoveContainer" containerID="cb0fb5bdff72faaddb4e50372de3849d17ed28ffe5d2c3dc0b1978a94b811869" Oct 31 01:45:03.200587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.200487 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" event={"ID":"19c558a3-547a-4d2d-ac43-a19be6e4493d","Type":"ContainerStarted","Data":"e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df"} Oct 31 01:45:03.203055 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.203019 5756 generic.go:356] "Generic (PLEG): container finished" podID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerID="8a024b36ecdba8ba1a390b1ce5971774c15405d519dd43a38728634d5b1da2e2" exitCode=60 Oct 31 01:45:03.203150 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.203049 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerDied","Data":"8a024b36ecdba8ba1a390b1ce5971774c15405d519dd43a38728634d5b1da2e2"} Oct 31 01:45:03.203312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.203292 5756 scope.go:122] "RemoveContainer" containerID="8a024b36ecdba8ba1a390b1ce5971774c15405d519dd43a38728634d5b1da2e2" Oct 31 01:45:03.414609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.414548 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" podStartSLOduration=3.4145301310000002 podStartE2EDuration="3.414530131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.414094056 +0000 UTC m=+2875.221061602" watchObservedRunningTime="2026-10-31 01:45:03.414530131 +0000 UTC m=+2875.221497679" Oct 31 01:45:03.870350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.870304 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" podStartSLOduration=3.870289781 podStartE2EDuration="3.870289781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.869650671 +0000 UTC m=+2875.676618214" watchObservedRunningTime="2026-10-31 01:45:03.870289781 +0000 UTC m=+2875.677257324" Oct 31 01:45:03.984194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:03.984144 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" podStartSLOduration=3.984130006 podStartE2EDuration="3.984130006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:03.981955726 +0000 UTC m=+2875.788923282" watchObservedRunningTime="2026-10-31 01:45:03.984130006 +0000 UTC m=+2875.791097548" Oct 31 01:45:04.212292 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.212259 5756 generic.go:356] "Generic (PLEG): container finished" podID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerID="a5e24b2cbe4fd914a8ab84d6e5c36c8fa509bb74a328785699f4b72b95b0f413" exitCode=60 Oct 31 01:45:04.212467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.212345 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" event={"ID":"b53321bd-0aff-4ef5-8073-3a767cbd0329","Type":"ContainerDied","Data":"a5e24b2cbe4fd914a8ab84d6e5c36c8fa509bb74a328785699f4b72b95b0f413"} Oct 31 01:45:04.212467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.212413 5756 scope.go:122] "RemoveContainer" containerID="95a1f18b81a621619287199d88106e60c239da6e6d913e6ae811901ae3ec7aa4" Oct 31 01:45:04.212707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.212690 5756 scope.go:122] "RemoveContainer" containerID="a5e24b2cbe4fd914a8ab84d6e5c36c8fa509bb74a328785699f4b72b95b0f413" Oct 31 01:45:04.213056 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:04.213033 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt_odh-ai-gateway-infra(b53321bd-0aff-4ef5-8073-3a767cbd0329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" Oct 31 01:45:04.214518 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.214490 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" event={"ID":"b544add2-4eaa-4fc5-aef6-80a1dd872786","Type":"ContainerStarted","Data":"ed71f29f0f9c02f3f8ec874a71cc491967882659d5f48ef7371b43560a754aa1"} Oct 31 01:45:04.216359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.216329 5756 generic.go:356] "Generic (PLEG): container finished" podID="77c6093f-61cc-4b3c-97df-776555390fd7" containerID="546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e" exitCode=60 Oct 31 01:45:04.216472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.216397 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" event={"ID":"77c6093f-61cc-4b3c-97df-776555390fd7","Type":"ContainerDied","Data":"546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e"} Oct 31 01:45:04.216671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.216654 5756 scope.go:122] "RemoveContainer" containerID="546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e" Oct 31 01:45:04.217021 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:04.216912 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t_odh-ai-gateway-infra(77c6093f-61cc-4b3c-97df-776555390fd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" Oct 31 01:45:04.218578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.218559 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" event={"ID":"9af9a4ba-e1ee-4481-89de-acb671353972","Type":"ContainerStarted","Data":"a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29"} Oct 31 01:45:04.220606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.220586 5756 generic.go:356] "Generic (PLEG): container finished" podID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerID="e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df" exitCode=60 Oct 31 01:45:04.220691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.220652 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" event={"ID":"19c558a3-547a-4d2d-ac43-a19be6e4493d","Type":"ContainerDied","Data":"e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df"} Oct 31 01:45:04.220876 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.220846 5756 scope.go:122] "RemoveContainer" containerID="e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df" Oct 31 01:45:04.221120 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:04.221096 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h_odh-ai-gateway-infra(19c558a3-547a-4d2d-ac43-a19be6e4493d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" Oct 31 01:45:04.222660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.222640 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerStarted","Data":"d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9"} Oct 31 01:45:04.224549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.224526 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" event={"ID":"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a","Type":"ContainerStarted","Data":"ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776"} Oct 31 01:45:04.226679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.226663 5756 scope.go:122] "RemoveContainer" containerID="ec1f424dacca1882ad7aa0bc681ba7554ae999e43f6edbecbe95d4a4b2dc8525" Oct 31 01:45:04.226767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.226684 5756 generic.go:356] "Generic (PLEG): container finished" podID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerID="18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2" exitCode=60 Oct 31 01:45:04.226825 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.226761 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" event={"ID":"377cfa2f-5923-4732-a87e-bedcc17090ba","Type":"ContainerDied","Data":"18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2"} Oct 31 01:45:04.226962 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.226946 5756 scope.go:122] "RemoveContainer" containerID="18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2" Oct 31 01:45:04.227173 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:04.227150 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4_odh-ai-gateway-infra(377cfa2f-5923-4732-a87e-bedcc17090ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" Oct 31 01:45:04.304668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.304646 5756 scope.go:122] "RemoveContainer" containerID="dd1212e23b30de5df75c83fb8675a60a982d453ea12b7aeeb10be644fd774ece" Oct 31 01:45:04.318330 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.318304 5756 scope.go:122] "RemoveContainer" containerID="2ac686646b339c0c6817f0283f2cbdfbdaabac0bf81d58a7773c4fe59d9a535a" Oct 31 01:45:04.775069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.774969 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" podStartSLOduration=4.774949887 podStartE2EDuration="4.774949887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:02.839320247 +0000 UTC m=+2874.646287790" watchObservedRunningTime="2026-10-31 01:45:04.774949887 +0000 UTC m=+2876.581917432" Oct 31 01:45:04.889025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:04.888971 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" podStartSLOduration=4.888958356 podStartE2EDuration="4.888958356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:04.886659572 +0000 UTC m=+2876.693627118" watchObservedRunningTime="2026-10-31 01:45:04.888958356 +0000 UTC m=+2876.695925899" Oct 31 01:45:05.112227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.112173 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" podStartSLOduration=5.112158969 podStartE2EDuration="5.112158969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:05.109694978 +0000 UTC m=+2876.916662522" watchObservedRunningTime="2026-10-31 01:45:05.112158969 +0000 UTC m=+2876.919126512" Oct 31 01:45:05.240052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.240015 5756 scope.go:122] "RemoveContainer" containerID="546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e" Oct 31 01:45:05.240375 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.240352 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t_odh-ai-gateway-infra(77c6093f-61cc-4b3c-97df-776555390fd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" Oct 31 01:45:05.241736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.241707 5756 generic.go:356] "Generic (PLEG): container finished" podID="9af9a4ba-e1ee-4481-89de-acb671353972" containerID="a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29" exitCode=60 Oct 31 01:45:05.241874 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.241790 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" event={"ID":"9af9a4ba-e1ee-4481-89de-acb671353972","Type":"ContainerDied","Data":"a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29"} Oct 31 01:45:05.241946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.241903 5756 scope.go:122] "RemoveContainer" containerID="cb0fb5bdff72faaddb4e50372de3849d17ed28ffe5d2c3dc0b1978a94b811869" Oct 31 01:45:05.242025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.242009 5756 scope.go:122] "RemoveContainer" containerID="a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29" Oct 31 01:45:05.242319 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.242292 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5_odh-ai-gateway-infra(9af9a4ba-e1ee-4481-89de-acb671353972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" Oct 31 01:45:05.243742 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.243726 5756 scope.go:122] "RemoveContainer" containerID="e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df" Oct 31 01:45:05.244025 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.244005 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h_odh-ai-gateway-infra(19c558a3-547a-4d2d-ac43-a19be6e4493d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" Oct 31 01:45:05.245389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.245370 5756 generic.go:356] "Generic (PLEG): container finished" podID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerID="d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9" exitCode=60 Oct 31 01:45:05.245487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.245440 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerDied","Data":"d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9"} Oct 31 01:45:05.245654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.245636 5756 scope.go:122] "RemoveContainer" containerID="d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9" Oct 31 01:45:05.245912 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.245891 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp_odh-ai-gateway-infra(4f7060ce-e966-4579-aa1d-964bef4a3efe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" Oct 31 01:45:05.247570 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.247548 5756 generic.go:356] "Generic (PLEG): container finished" podID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerID="ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776" exitCode=60 Oct 31 01:45:05.247674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.247653 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" event={"ID":"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a","Type":"ContainerDied","Data":"ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776"} Oct 31 01:45:05.247800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.247785 5756 scope.go:122] "RemoveContainer" containerID="ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776" Oct 31 01:45:05.247996 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.247980 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw_odh-ai-gateway-infra(ebd19de2-2772-40fb-ab58-2e35f5ef3c2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" Oct 31 01:45:05.249625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.249607 5756 scope.go:122] "RemoveContainer" containerID="18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2" Oct 31 01:45:05.249854 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.249833 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4_odh-ai-gateway-infra(377cfa2f-5923-4732-a87e-bedcc17090ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" Oct 31 01:45:05.251406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.251388 5756 scope.go:122] "RemoveContainer" containerID="a5e24b2cbe4fd914a8ab84d6e5c36c8fa509bb74a328785699f4b72b95b0f413" Oct 31 01:45:05.251648 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.251623 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt_odh-ai-gateway-infra(b53321bd-0aff-4ef5-8073-3a767cbd0329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" Oct 31 01:45:05.253116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.253094 5756 generic.go:356] "Generic (PLEG): container finished" podID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerID="ed71f29f0f9c02f3f8ec874a71cc491967882659d5f48ef7371b43560a754aa1" exitCode=60 Oct 31 01:45:05.253187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.253141 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" event={"ID":"b544add2-4eaa-4fc5-aef6-80a1dd872786","Type":"ContainerDied","Data":"ed71f29f0f9c02f3f8ec874a71cc491967882659d5f48ef7371b43560a754aa1"} Oct 31 01:45:05.253361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.253349 5756 scope.go:122] "RemoveContainer" containerID="ed71f29f0f9c02f3f8ec874a71cc491967882659d5f48ef7371b43560a754aa1" Oct 31 01:45:05.253542 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:05.253519 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb_odh-ai-gateway-infra(b544add2-4eaa-4fc5-aef6-80a1dd872786)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" Oct 31 01:45:05.256533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.256516 5756 scope.go:122] "RemoveContainer" containerID="8a024b36ecdba8ba1a390b1ce5971774c15405d519dd43a38728634d5b1da2e2" Oct 31 01:45:05.271104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.271085 5756 scope.go:122] "RemoveContainer" containerID="6fd04c53c9ecc6eb6ca2297207e96f716fb3de0ecdbc96854eaebae345ffd624" Oct 31 01:45:05.284241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:05.284222 5756 scope.go:122] "RemoveContainer" containerID="f27bfcb8a190968e787900be7f346e4ad0abb9c84ffe847969581feb354233d6" Oct 31 01:45:06.262041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:06.262013 5756 scope.go:122] "RemoveContainer" containerID="ed71f29f0f9c02f3f8ec874a71cc491967882659d5f48ef7371b43560a754aa1" Oct 31 01:45:06.262596 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:06.262341 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb_odh-ai-gateway-infra(b544add2-4eaa-4fc5-aef6-80a1dd872786)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" Oct 31 01:45:06.263785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:06.263753 5756 scope.go:122] "RemoveContainer" containerID="a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29" Oct 31 01:45:06.264073 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:06.264053 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5_odh-ai-gateway-infra(9af9a4ba-e1ee-4481-89de-acb671353972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" Oct 31 01:45:06.265408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:06.265389 5756 scope.go:122] "RemoveContainer" containerID="d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9" Oct 31 01:45:06.265659 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:06.265637 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp_odh-ai-gateway-infra(4f7060ce-e966-4579-aa1d-964bef4a3efe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" Oct 31 01:45:06.266930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:06.266913 5756 scope.go:122] "RemoveContainer" containerID="ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776" Oct 31 01:45:06.267128 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:06.267111 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw_odh-ai-gateway-infra(ebd19de2-2772-40fb-ab58-2e35f5ef3c2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" Oct 31 01:45:08.665237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:08.665205 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj"] Oct 31 01:45:08.772732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:08.772698 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf"] Oct 31 01:45:10.101137 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.101107 5756 scope.go:122] "RemoveContainer" containerID="8542bbcb244610c133a4467552909c4cdda8c8e9e2359b06e23cc1ed95ce9b47" Oct 31 01:45:10.113581 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.113557 5756 scope.go:122] "RemoveContainer" containerID="a2d14b4ff755d2995b27b7a24db65434303cd7d9f7275e5363880476466afb1e" Oct 31 01:45:10.123910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.123886 5756 scope.go:122] "RemoveContainer" containerID="611046d1df284604b6b56438b04760bacf3388b2125be3779535f5187a2ec589" Oct 31 01:45:10.686547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.686515 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb"] Oct 31 01:45:10.857096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.857060 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:10.930838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.930773 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9\" (UniqueName: \"kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9\") pod \"b544add2-4eaa-4fc5-aef6-80a1dd872786\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " Oct 31 01:45:10.931068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.931007 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca\") pod \"b544add2-4eaa-4fc5-aef6-80a1dd872786\" (UID: \"b544add2-4eaa-4fc5-aef6-80a1dd872786\") " Oct 31 01:45:10.931411 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.931383 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca" pod "b544add2-4eaa-4fc5-aef6-80a1dd872786" (UID: "b544add2-4eaa-4fc5-aef6-80a1dd872786"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:10.933220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:10.933189 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9" pod "b544add2-4eaa-4fc5-aef6-80a1dd872786" (UID: "b544add2-4eaa-4fc5-aef6-80a1dd872786"). InnerVolumeSpecName "kube-api-access-8zvc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:11.033032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.032998 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b544add2-4eaa-4fc5-aef6-80a1dd872786-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:11.033032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.033029 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zvc9\" (UniqueName: \"kubernetes.io/projected/b544add2-4eaa-4fc5-aef6-80a1dd872786-kube-api-access-8zvc9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:11.305950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.305918 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" Oct 31 01:45:11.306435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.305968 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb" event={"ID":"b544add2-4eaa-4fc5-aef6-80a1dd872786","Type":"ContainerDied","Data":"8b390324424496ae4ac58f0795e868a1e38c2f40087e39c28813cc9bfb75c80e"} Oct 31 01:45:11.306435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.306026 5756 scope.go:122] "RemoveContainer" containerID="ed71f29f0f9c02f3f8ec874a71cc491967882659d5f48ef7371b43560a754aa1" Oct 31 01:45:11.468849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.468818 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb"] Oct 31 01:45:11.471250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:11.471228 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-66100b-29890185-4f6gb"] Oct 31 01:45:12.311659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:12.311627 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" path="/var/lib/kubelet/pods/b544add2-4eaa-4fc5-aef6-80a1dd872786/volumes" Oct 31 01:45:13.666437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:13.666403 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" podUID="7620a66e-79fe-40b1-b4aa-56eaf7574e5d" containerName="payload-processing" containerID="cri-o://72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf" gracePeriod=25 Oct 31 01:45:13.773452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:13.773412 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" podUID="b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" containerName="payload-pre-processing" containerID="cri-o://6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4" gracePeriod=25 Oct 31 01:45:13.943265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:13.943241 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:45:14.045481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.045454 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:45:14.068136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.068108 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277\" (UniqueName: \"kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277\") pod \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " Oct 31 01:45:14.068289 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.068211 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume\") pod \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\" (UID: \"7620a66e-79fe-40b1-b4aa-56eaf7574e5d\") " Oct 31 01:45:14.068633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.068610 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume" pod "7620a66e-79fe-40b1-b4aa-56eaf7574e5d" (UID: "7620a66e-79fe-40b1-b4aa-56eaf7574e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:14.070315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.070296 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277" pod "7620a66e-79fe-40b1-b4aa-56eaf7574e5d" (UID: "7620a66e-79fe-40b1-b4aa-56eaf7574e5d"). InnerVolumeSpecName "kube-api-access-gk277". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:14.169397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.169369 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m\" (UniqueName: \"kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m\") pod \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " Oct 31 01:45:14.169560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.169482 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume\") pod \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\" (UID: \"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f\") " Oct 31 01:45:14.169750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.169735 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk277\" (UniqueName: \"kubernetes.io/projected/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-kube-api-access-gk277\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:14.169750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.169750 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7620a66e-79fe-40b1-b4aa-56eaf7574e5d-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:14.169925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.169902 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume" pod "b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" (UID: "b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:14.171584 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.171542 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m" pod "b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" (UID: "b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f"). InnerVolumeSpecName "kube-api-access-s9d7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:14.270801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.270763 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9d7m\" (UniqueName: \"kubernetes.io/projected/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-kube-api-access-s9d7m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:14.270801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.270791 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:14.328013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.327985 5756 generic.go:356] "Generic (PLEG): container finished" podID="7620a66e-79fe-40b1-b4aa-56eaf7574e5d" containerID="72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf" exitCode=0 Oct 31 01:45:14.328152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.328026 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" Oct 31 01:45:14.328152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.328069 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" event={"ID":"7620a66e-79fe-40b1-b4aa-56eaf7574e5d","Type":"ContainerDied","Data":"72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf"} Oct 31 01:45:14.328152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.328128 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj" event={"ID":"7620a66e-79fe-40b1-b4aa-56eaf7574e5d","Type":"ContainerDied","Data":"eebe53d8697c489a98d7525c4b25585e931e9b4d7613634ebe36927204cc76b8"} Oct 31 01:45:14.328307 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.328156 5756 scope.go:122] "RemoveContainer" containerID="72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf" Oct 31 01:45:14.329770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.329743 5756 generic.go:356] "Generic (PLEG): container finished" podID="b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" containerID="6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4" exitCode=0 Oct 31 01:45:14.329906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.329789 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" Oct 31 01:45:14.329906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.329822 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" event={"ID":"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f","Type":"ContainerDied","Data":"6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4"} Oct 31 01:45:14.329906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.329875 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf" event={"ID":"b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f","Type":"ContainerDied","Data":"a4fab2adebf6460fe1cb4b1238fd5c88568de0054dd3800b6f52f2a2ecab4fe6"} Oct 31 01:45:14.339079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.339054 5756 scope.go:122] "RemoveContainer" containerID="72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf" Oct 31 01:45:14.339397 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:14.339373 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf\": container with ID starting with 72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf not found: ID does not exist" containerID="72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf" Oct 31 01:45:14.339509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.339407 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf"} err="failed to get container status \"72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf\": rpc error: code = NotFound desc = could not find container \"72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf\": container with ID starting with 72ab3f2a626c52e433deac1e72c44eebc1cfccdb6c02622250bb934a9b854cbf not found: ID does not exist" Oct 31 01:45:14.339509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.339431 5756 scope.go:122] "RemoveContainer" containerID="6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4" Oct 31 01:45:14.349054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.349035 5756 scope.go:122] "RemoveContainer" containerID="6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4" Oct 31 01:45:14.349402 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:14.349383 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4\": container with ID starting with 6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4 not found: ID does not exist" containerID="6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4" Oct 31 01:45:14.349478 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.349408 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4"} err="failed to get container status \"6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4\": rpc error: code = NotFound desc = could not find container \"6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4\": container with ID starting with 6009cfb6f12386a2a6140bba049ae4d3fea85c8f235d974a7b232ad8df87c2a4 not found: ID does not exist" Oct 31 01:45:14.490694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.490580 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj"] Oct 31 01:45:14.493561 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.493533 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-66100b-855b664dd9-ljdvj"] Oct 31 01:45:14.575329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575297 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m"] Oct 31 01:45:14.575818 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575805 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" containerName="payload-pre-processing" Oct 31 01:45:14.575886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575820 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" containerName="payload-pre-processing" Oct 31 01:45:14.575886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575830 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerName="cleanup" Oct 31 01:45:14.575886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575835 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerName="cleanup" Oct 31 01:45:14.575886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575885 5756 cpu_manager.go:414] "containerMap: removing container" podUID="7620a66e-79fe-40b1-b4aa-56eaf7574e5d" containerName="payload-processing" Oct 31 01:45:14.576043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575890 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7620a66e-79fe-40b1-b4aa-56eaf7574e5d" containerName="payload-processing" Oct 31 01:45:14.576043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575984 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" containerName="payload-pre-processing" Oct 31 01:45:14.576043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.575993 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerName="cleanup" Oct 31 01:45:14.576043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.576002 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="7620a66e-79fe-40b1-b4aa-56eaf7574e5d" containerName="payload-processing" Oct 31 01:45:14.580562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.580541 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.601704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.601673 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf"] Oct 31 01:45:14.609067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.609045 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-66100b-7489ddbc79-kbxkf"] Oct 31 01:45:14.633609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.633582 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:45:14.635758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.635739 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m"] Oct 31 01:45:14.674918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.674884 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.675316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.675002 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s92n\" (UniqueName: \"kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.675316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.675038 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.776709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.776623 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s92n\" (UniqueName: \"kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.776709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.776675 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.776981 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.776832 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.777403 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.777374 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.779435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.779409 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.833549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.833519 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s92n\" (UniqueName: \"kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n\") pod \"maas-api-revoke-sub-c22e2353-v725m\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:14.893660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:14.893631 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:15.268653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:15.268616 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m"] Oct 31 01:45:15.268828 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:15.268808 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3c7885_9bc0_4daa_8381_0fb2f83b9831.slice/crio-98a7cc964948d11858fe9027e6c65a7a2fb995e663ec4c4d94e2e578de86c2e7 WatchSource:0}: Error finding container 98a7cc964948d11858fe9027e6c65a7a2fb995e663ec4c4d94e2e578de86c2e7: Status 404 returned error can't find the container with id 98a7cc964948d11858fe9027e6c65a7a2fb995e663ec4c4d94e2e578de86c2e7 Oct 31 01:45:15.304166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:15.304141 5756 scope.go:122] "RemoveContainer" containerID="e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df" Oct 31 01:45:15.304294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:15.304232 5756 scope.go:122] "RemoveContainer" containerID="d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9" Oct 31 01:45:15.339243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:15.339218 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" event={"ID":"0b3c7885-9bc0-4daa-8381-0fb2f83b9831","Type":"ContainerStarted","Data":"98a7cc964948d11858fe9027e6c65a7a2fb995e663ec4c4d94e2e578de86c2e7"} Oct 31 01:45:16.304502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.304467 5756 scope.go:122] "RemoveContainer" containerID="18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2" Oct 31 01:45:16.311826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.311799 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7620a66e-79fe-40b1-b4aa-56eaf7574e5d" path="/var/lib/kubelet/pods/7620a66e-79fe-40b1-b4aa-56eaf7574e5d/volumes" Oct 31 01:45:16.312222 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.312207 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f" path="/var/lib/kubelet/pods/b3d7af40-120d-4e75-b2b1-63a6e0c5ad6f/volumes" Oct 31 01:45:16.349759 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.349728 5756 generic.go:356] "Generic (PLEG): container finished" podID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerID="42200f4269e27686aa108f0ca97e7709a79e8cfa6fe8687e23b3998b4ed0ab2d" exitCode=60 Oct 31 01:45:16.349894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.349806 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" event={"ID":"19c558a3-547a-4d2d-ac43-a19be6e4493d","Type":"ContainerDied","Data":"42200f4269e27686aa108f0ca97e7709a79e8cfa6fe8687e23b3998b4ed0ab2d"} Oct 31 01:45:16.349894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.349883 5756 scope.go:122] "RemoveContainer" containerID="e128b0b36706ba99523bd9db1a5998c612dc4330ba9cefce6be8e57cc33d20df" Oct 31 01:45:16.350303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.350279 5756 scope.go:122] "RemoveContainer" containerID="42200f4269e27686aa108f0ca97e7709a79e8cfa6fe8687e23b3998b4ed0ab2d" Oct 31 01:45:16.350897 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:16.350689 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h_odh-ai-gateway-infra(19c558a3-547a-4d2d-ac43-a19be6e4493d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" Oct 31 01:45:16.354071 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.354042 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerStarted","Data":"cab28dd833bd14bd74320751625425108792b34a21508572e272554c2f041cf8"} Oct 31 01:45:16.357062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.357040 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" event={"ID":"0b3c7885-9bc0-4daa-8381-0fb2f83b9831","Type":"ContainerStarted","Data":"163655be0316a383d83dfe5981c7ff05e7ff258b7a4a8a90024cb79296cd713e"} Oct 31 01:45:16.574081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:16.573984 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" podStartSLOduration=2.573970869 podStartE2EDuration="2.573970869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:16.571604018 +0000 UTC m=+2888.378571559" watchObservedRunningTime="2026-10-31 01:45:16.573970869 +0000 UTC m=+2888.380938413" Oct 31 01:45:17.304280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.304252 5756 scope.go:122] "RemoveContainer" containerID="ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776" Oct 31 01:45:17.304469 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.304423 5756 scope.go:122] "RemoveContainer" containerID="546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e" Oct 31 01:45:17.304617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.304598 5756 scope.go:122] "RemoveContainer" containerID="a5e24b2cbe4fd914a8ab84d6e5c36c8fa509bb74a328785699f4b72b95b0f413" Oct 31 01:45:17.368604 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.368578 5756 generic.go:356] "Generic (PLEG): container finished" podID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerID="cab28dd833bd14bd74320751625425108792b34a21508572e272554c2f041cf8" exitCode=60 Oct 31 01:45:17.368767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.368650 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerDied","Data":"cab28dd833bd14bd74320751625425108792b34a21508572e272554c2f041cf8"} Oct 31 01:45:17.368767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.368714 5756 scope.go:122] "RemoveContainer" containerID="d99f874131a0738c485f5b0888f68a65fc035d94420d1bf6c9393da2ee8ef2b9" Oct 31 01:45:17.369116 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.369094 5756 scope.go:122] "RemoveContainer" containerID="cab28dd833bd14bd74320751625425108792b34a21508572e272554c2f041cf8" Oct 31 01:45:17.369415 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:17.369394 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp_odh-ai-gateway-infra(4f7060ce-e966-4579-aa1d-964bef4a3efe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" Oct 31 01:45:17.370711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.370691 5756 generic.go:356] "Generic (PLEG): container finished" podID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerID="d483645f66f1352c88f950298792c4c53caaefb521d80158e4536caf5a768a17" exitCode=60 Oct 31 01:45:17.370801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.370764 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" event={"ID":"377cfa2f-5923-4732-a87e-bedcc17090ba","Type":"ContainerDied","Data":"d483645f66f1352c88f950298792c4c53caaefb521d80158e4536caf5a768a17"} Oct 31 01:45:17.371168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.371153 5756 scope.go:122] "RemoveContainer" containerID="d483645f66f1352c88f950298792c4c53caaefb521d80158e4536caf5a768a17" Oct 31 01:45:17.371430 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:17.371406 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4_odh-ai-gateway-infra(377cfa2f-5923-4732-a87e-bedcc17090ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" Oct 31 01:45:17.372379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.372350 5756 generic.go:356] "Generic (PLEG): container finished" podID="0b3c7885-9bc0-4daa-8381-0fb2f83b9831" containerID="163655be0316a383d83dfe5981c7ff05e7ff258b7a4a8a90024cb79296cd713e" exitCode=0 Oct 31 01:45:17.372572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.372554 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" event={"ID":"0b3c7885-9bc0-4daa-8381-0fb2f83b9831","Type":"ContainerDied","Data":"163655be0316a383d83dfe5981c7ff05e7ff258b7a4a8a90024cb79296cd713e"} Oct 31 01:45:17.383261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.383244 5756 scope.go:122] "RemoveContainer" containerID="18fae2b317f476ca498bb0840778d7c879d5453917d6b7768ae6cc21c04d79d2" Oct 31 01:45:17.470732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.470701 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h"] Oct 31 01:45:17.620448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.620424 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:17.690995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.690959 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp"] Oct 31 01:45:17.711787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.711750 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca\") pod \"19c558a3-547a-4d2d-ac43-a19be6e4493d\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " Oct 31 01:45:17.712264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.712241 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca" pod "19c558a3-547a-4d2d-ac43-a19be6e4493d" (UID: "19c558a3-547a-4d2d-ac43-a19be6e4493d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:17.712332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.712298 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl\" (UniqueName: \"kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl\") pod \"19c558a3-547a-4d2d-ac43-a19be6e4493d\" (UID: \"19c558a3-547a-4d2d-ac43-a19be6e4493d\") " Oct 31 01:45:17.712694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.712681 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19c558a3-547a-4d2d-ac43-a19be6e4493d-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:17.728673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.728641 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl" pod "19c558a3-547a-4d2d-ac43-a19be6e4493d" (UID: "19c558a3-547a-4d2d-ac43-a19be6e4493d"). InnerVolumeSpecName "kube-api-access-zlmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:17.813852 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:17.813819 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlmxl\" (UniqueName: \"kubernetes.io/projected/19c558a3-547a-4d2d-ac43-a19be6e4493d-kube-api-access-zlmxl\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.382102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.382066 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" event={"ID":"b53321bd-0aff-4ef5-8073-3a767cbd0329","Type":"ContainerStarted","Data":"1ae0607c557dece87c66c8c4b2d2ab5eada9728efa9b5ac3c9f5204c192767bf"} Oct 31 01:45:18.384314 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.384285 5756 generic.go:356] "Generic (PLEG): container finished" podID="77c6093f-61cc-4b3c-97df-776555390fd7" containerID="bc8fa87aa1bac34159f444b0fe68cb8ecf7fc35b7632034ef2572c411c19695a" exitCode=60 Oct 31 01:45:18.384459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.384356 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" event={"ID":"77c6093f-61cc-4b3c-97df-776555390fd7","Type":"ContainerDied","Data":"bc8fa87aa1bac34159f444b0fe68cb8ecf7fc35b7632034ef2572c411c19695a"} Oct 31 01:45:18.384459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.384417 5756 scope.go:122] "RemoveContainer" containerID="546c9ffb166cc3468ad8e3664f19a3d67d0cc5d3f46b518a700dd18d6a887b4e" Oct 31 01:45:18.384834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.384815 5756 scope.go:122] "RemoveContainer" containerID="bc8fa87aa1bac34159f444b0fe68cb8ecf7fc35b7632034ef2572c411c19695a" Oct 31 01:45:18.385162 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:18.385133 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t_odh-ai-gateway-infra(77c6093f-61cc-4b3c-97df-776555390fd7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" Oct 31 01:45:18.386130 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.386113 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" Oct 31 01:45:18.386240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.386118 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h" event={"ID":"19c558a3-547a-4d2d-ac43-a19be6e4493d","Type":"ContainerDied","Data":"496d0f08c91184586e95912832d64af5c39583b08215cf8efad59abee6fd2e0f"} Oct 31 01:45:18.390156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.390133 5756 generic.go:356] "Generic (PLEG): container finished" podID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerID="7db1b4313babbb2fcee53ce86aad686ba69ca3e45c5941ffebf482ded3d54e58" exitCode=60 Oct 31 01:45:18.390254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.390216 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" event={"ID":"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a","Type":"ContainerDied","Data":"7db1b4313babbb2fcee53ce86aad686ba69ca3e45c5941ffebf482ded3d54e58"} Oct 31 01:45:18.390598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.390581 5756 scope.go:122] "RemoveContainer" containerID="7db1b4313babbb2fcee53ce86aad686ba69ca3e45c5941ffebf482ded3d54e58" Oct 31 01:45:18.390989 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:18.390877 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw_odh-ai-gateway-infra(ebd19de2-2772-40fb-ab58-2e35f5ef3c2a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" Oct 31 01:45:18.547912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.547889 5756 scope.go:122] "RemoveContainer" containerID="42200f4269e27686aa108f0ca97e7709a79e8cfa6fe8687e23b3998b4ed0ab2d" Oct 31 01:45:18.569282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.569261 5756 scope.go:122] "RemoveContainer" containerID="ae0c205ec5466eaaa90cb362097350a85db51f238b62608e99ee2b00c4d4a776" Oct 31 01:45:18.575163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.575137 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:18.586247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.586230 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:18.603111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.603085 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4"] Oct 31 01:45:18.735699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.735674 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:18.740904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.740880 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca\") pod \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " Oct 31 01:45:18.741032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.740938 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n\" (UniqueName: \"kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n\") pod \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " Oct 31 01:45:18.741032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.740966 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca\") pod \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\" (UID: \"0b3c7885-9bc0-4daa-8381-0fb2f83b9831\") " Oct 31 01:45:18.741143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.741032 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw\" (UniqueName: \"kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw\") pod \"4f7060ce-e966-4579-aa1d-964bef4a3efe\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " Oct 31 01:45:18.741143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.741115 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca\") pod \"4f7060ce-e966-4579-aa1d-964bef4a3efe\" (UID: \"4f7060ce-e966-4579-aa1d-964bef4a3efe\") " Oct 31 01:45:18.741453 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.741421 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca" pod "0b3c7885-9bc0-4daa-8381-0fb2f83b9831" (UID: "0b3c7885-9bc0-4daa-8381-0fb2f83b9831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:18.741572 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.741495 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca" pod "4f7060ce-e966-4579-aa1d-964bef4a3efe" (UID: "4f7060ce-e966-4579-aa1d-964bef4a3efe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:18.741840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.741728 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7060ce-e966-4579-aa1d-964bef4a3efe-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.741840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.741751 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.743152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.743126 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca" pod "0b3c7885-9bc0-4daa-8381-0fb2f83b9831" (UID: "0b3c7885-9bc0-4daa-8381-0fb2f83b9831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:45:18.743152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.743128 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n" pod "0b3c7885-9bc0-4daa-8381-0fb2f83b9831" (UID: "0b3c7885-9bc0-4daa-8381-0fb2f83b9831"). InnerVolumeSpecName "kube-api-access-9s92n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:18.743272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.743260 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw" pod "4f7060ce-e966-4579-aa1d-964bef4a3efe" (UID: "4f7060ce-e966-4579-aa1d-964bef4a3efe"). InnerVolumeSpecName "kube-api-access-k8llw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:18.842669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.842591 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9\" (UniqueName: \"kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9\") pod \"377cfa2f-5923-4732-a87e-bedcc17090ba\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " Oct 31 01:45:18.842831 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.842721 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca\") pod \"377cfa2f-5923-4732-a87e-bedcc17090ba\" (UID: \"377cfa2f-5923-4732-a87e-bedcc17090ba\") " Oct 31 01:45:18.843042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.843026 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.843042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.843042 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s92n\" (UniqueName: \"kubernetes.io/projected/0b3c7885-9bc0-4daa-8381-0fb2f83b9831-kube-api-access-9s92n\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.843193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.843052 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8llw\" (UniqueName: \"kubernetes.io/projected/4f7060ce-e966-4579-aa1d-964bef4a3efe-kube-api-access-k8llw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.843193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.843121 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca" pod "377cfa2f-5923-4732-a87e-bedcc17090ba" (UID: "377cfa2f-5923-4732-a87e-bedcc17090ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:18.844839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.844818 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9" pod "377cfa2f-5923-4732-a87e-bedcc17090ba" (UID: "377cfa2f-5923-4732-a87e-bedcc17090ba"). InnerVolumeSpecName "kube-api-access-sqbm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:18.943672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.943640 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbm9\" (UniqueName: \"kubernetes.io/projected/377cfa2f-5923-4732-a87e-bedcc17090ba-kube-api-access-sqbm9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.943672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.943670 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/377cfa2f-5923-4732-a87e-bedcc17090ba-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:18.991662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.991627 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h"] Oct 31 01:45:18.995680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:18.995654 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-ec695b-29890185-qxw2h"] Oct 31 01:45:19.304937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.304912 5756 scope.go:122] "RemoveContainer" containerID="a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29" Oct 31 01:45:19.402472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.402437 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" event={"ID":"4f7060ce-e966-4579-aa1d-964bef4a3efe","Type":"ContainerDied","Data":"3f885c16ce71fb78a230cfbbaab289248189d4fe0c9af479f97cedad68d8f5bc"} Oct 31 01:45:19.402472 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.402468 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp" Oct 31 01:45:19.403030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.402501 5756 scope.go:122] "RemoveContainer" containerID="cab28dd833bd14bd74320751625425108792b34a21508572e272554c2f041cf8" Oct 31 01:45:19.405806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.405784 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" Oct 31 01:45:19.405946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.405788 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4" event={"ID":"377cfa2f-5923-4732-a87e-bedcc17090ba","Type":"ContainerDied","Data":"2e784bd737fa4e5647ed4f3b9ac8eb01ff96601b8b7068c21990e036dc927142"} Oct 31 01:45:19.407394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.407374 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" Oct 31 01:45:19.407480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.407397 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m" event={"ID":"0b3c7885-9bc0-4daa-8381-0fb2f83b9831","Type":"ContainerDied","Data":"98a7cc964948d11858fe9027e6c65a7a2fb995e663ec4c4d94e2e578de86c2e7"} Oct 31 01:45:19.407480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.407472 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98a7cc964948d11858fe9027e6c65a7a2fb995e663ec4c4d94e2e578de86c2e7" Oct 31 01:45:19.409660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.409636 5756 generic.go:356] "Generic (PLEG): container finished" podID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerID="1ae0607c557dece87c66c8c4b2d2ab5eada9728efa9b5ac3c9f5204c192767bf" exitCode=60 Oct 31 01:45:19.409760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.409671 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" event={"ID":"b53321bd-0aff-4ef5-8073-3a767cbd0329","Type":"ContainerDied","Data":"1ae0607c557dece87c66c8c4b2d2ab5eada9728efa9b5ac3c9f5204c192767bf"} Oct 31 01:45:19.410111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.410093 5756 scope.go:122] "RemoveContainer" containerID="1ae0607c557dece87c66c8c4b2d2ab5eada9728efa9b5ac3c9f5204c192767bf" Oct 31 01:45:19.410389 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:19.410367 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt_odh-ai-gateway-infra(b53321bd-0aff-4ef5-8073-3a767cbd0329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" Oct 31 01:45:19.414117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.414100 5756 scope.go:122] "RemoveContainer" containerID="d483645f66f1352c88f950298792c4c53caaefb521d80158e4536caf5a768a17" Oct 31 01:45:19.425539 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.425520 5756 scope.go:122] "RemoveContainer" containerID="a5e24b2cbe4fd914a8ab84d6e5c36c8fa509bb74a328785699f4b72b95b0f413" Oct 31 01:45:19.504265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.504190 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt"] Oct 31 01:45:19.621255 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.621205 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t"] Oct 31 01:45:19.731070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.731035 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw"] Oct 31 01:45:19.737920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.737889 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4"] Oct 31 01:45:19.751515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.751469 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-acd385-29890185-4zgf4"] Oct 31 01:45:19.771895 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.771819 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:19.851705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.851675 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca\") pod \"77c6093f-61cc-4b3c-97df-776555390fd7\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " Oct 31 01:45:19.851907 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.851782 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn\" (UniqueName: \"kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn\") pod \"77c6093f-61cc-4b3c-97df-776555390fd7\" (UID: \"77c6093f-61cc-4b3c-97df-776555390fd7\") " Oct 31 01:45:19.852296 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.852265 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca" pod "77c6093f-61cc-4b3c-97df-776555390fd7" (UID: "77c6093f-61cc-4b3c-97df-776555390fd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:19.854358 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.854336 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn" pod "77c6093f-61cc-4b3c-97df-776555390fd7" (UID: "77c6093f-61cc-4b3c-97df-776555390fd7"). InnerVolumeSpecName "kube-api-access-j6grn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:19.858892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.858856 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp"] Oct 31 01:45:19.869086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.869062 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-97b317-29890185-cwxdp"] Oct 31 01:45:19.875089 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.875066 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:19.952646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.952617 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm\" (UniqueName: \"kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm\") pod \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " Oct 31 01:45:19.952812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.952652 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca\") pod \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\" (UID: \"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a\") " Oct 31 01:45:19.953087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.953064 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca" pod "ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" (UID: "ebd19de2-2772-40fb-ab58-2e35f5ef3c2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:19.953171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.953156 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c6093f-61cc-4b3c-97df-776555390fd7-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:19.953228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.953176 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:19.953228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.953186 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6grn\" (UniqueName: \"kubernetes.io/projected/77c6093f-61cc-4b3c-97df-776555390fd7-kube-api-access-j6grn\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:19.954905 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:19.954884 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm" pod "ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" (UID: "ebd19de2-2772-40fb-ab58-2e35f5ef3c2a"). InnerVolumeSpecName "kube-api-access-wg4jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:20.053773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.053737 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg4jm\" (UniqueName: \"kubernetes.io/projected/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a-kube-api-access-wg4jm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:20.311343 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.311271 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" path="/var/lib/kubelet/pods/19c558a3-547a-4d2d-ac43-a19be6e4493d/volumes" Oct 31 01:45:20.311778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.311763 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" path="/var/lib/kubelet/pods/377cfa2f-5923-4732-a87e-bedcc17090ba/volumes" Oct 31 01:45:20.312209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.312196 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" path="/var/lib/kubelet/pods/4f7060ce-e966-4579-aa1d-964bef4a3efe/volumes" Oct 31 01:45:20.421162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.421134 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" Oct 31 01:45:20.421162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.421145 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t" event={"ID":"77c6093f-61cc-4b3c-97df-776555390fd7","Type":"ContainerDied","Data":"ec1723c21cbddc295c389f422d9286d3a8c4af7ebbec95ff7d15007c2ff4c629"} Oct 31 01:45:20.421639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.421198 5756 scope.go:122] "RemoveContainer" containerID="bc8fa87aa1bac34159f444b0fe68cb8ecf7fc35b7632034ef2572c411c19695a" Oct 31 01:45:20.423542 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.423513 5756 generic.go:356] "Generic (PLEG): container finished" podID="9af9a4ba-e1ee-4481-89de-acb671353972" containerID="98a040a8c7995fd8717d47f3a47ea8a36944e79690d7e9f7f0fc795e514e2c61" exitCode=60 Oct 31 01:45:20.423666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.423589 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" event={"ID":"9af9a4ba-e1ee-4481-89de-acb671353972","Type":"ContainerDied","Data":"98a040a8c7995fd8717d47f3a47ea8a36944e79690d7e9f7f0fc795e514e2c61"} Oct 31 01:45:20.424059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.424033 5756 scope.go:122] "RemoveContainer" containerID="98a040a8c7995fd8717d47f3a47ea8a36944e79690d7e9f7f0fc795e514e2c61" Oct 31 01:45:20.424361 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:20.424338 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5_odh-ai-gateway-infra(9af9a4ba-e1ee-4481-89de-acb671353972)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" Oct 31 01:45:20.426377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.426355 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" event={"ID":"ebd19de2-2772-40fb-ab58-2e35f5ef3c2a","Type":"ContainerDied","Data":"3a6f12cc57b7dacf5241de2c2271765c1c394d50bc3bb604e242320c1afd268f"} Oct 31 01:45:20.426506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.426384 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw" Oct 31 01:45:20.432285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.432255 5756 scope.go:122] "RemoveContainer" containerID="a49d55bd8f532ebdf4d911c1f1aca5ab2d8dc28ad291d7196f537fbda5704b29" Oct 31 01:45:20.447069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.447044 5756 scope.go:122] "RemoveContainer" containerID="7db1b4313babbb2fcee53ce86aad686ba69ca3e45c5941ffebf482ded3d54e58" Oct 31 01:45:20.556555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.556527 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:20.557599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.557582 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw\" (UniqueName: \"kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw\") pod \"b53321bd-0aff-4ef5-8073-3a767cbd0329\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " Oct 31 01:45:20.557663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.557650 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca\") pod \"b53321bd-0aff-4ef5-8073-3a767cbd0329\" (UID: \"b53321bd-0aff-4ef5-8073-3a767cbd0329\") " Oct 31 01:45:20.557995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.557969 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca" pod "b53321bd-0aff-4ef5-8073-3a767cbd0329" (UID: "b53321bd-0aff-4ef5-8073-3a767cbd0329"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:20.559746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.559730 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw" pod "b53321bd-0aff-4ef5-8073-3a767cbd0329" (UID: "b53321bd-0aff-4ef5-8073-3a767cbd0329"). InnerVolumeSpecName "kube-api-access-rmdkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:20.642172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.642133 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t"] Oct 31 01:45:20.647821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.647793 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-26fbc4-29890185-f479t"] Oct 31 01:45:20.659102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.659079 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmdkw\" (UniqueName: \"kubernetes.io/projected/b53321bd-0aff-4ef5-8073-3a767cbd0329-kube-api-access-rmdkw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:20.659102 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.659102 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b53321bd-0aff-4ef5-8073-3a767cbd0329-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:20.754192 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.754154 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw"] Oct 31 01:45:20.760024 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:20.759996 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-25b9ff-29890185-lhffw"] Oct 31 01:45:21.436949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:21.436920 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" Oct 31 01:45:21.436949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:21.436923 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt" event={"ID":"b53321bd-0aff-4ef5-8073-3a767cbd0329","Type":"ContainerDied","Data":"7bd64314db922788ed42685d683aae337cd378dae70695e2adc1d4d0c06d8454"} Oct 31 01:45:21.437467 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:21.436994 5756 scope.go:122] "RemoveContainer" containerID="1ae0607c557dece87c66c8c4b2d2ab5eada9728efa9b5ac3c9f5204c192767bf" Oct 31 01:45:21.600013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:21.599980 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt"] Oct 31 01:45:21.604921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:21.604897 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-76aba6-29890185-rqvjt"] Oct 31 01:45:21.882057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:21.882021 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5"] Oct 31 01:45:22.020530 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.020507 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:22.077488 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.077462 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca\") pod \"9af9a4ba-e1ee-4481-89de-acb671353972\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " Oct 31 01:45:22.077651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.077521 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z\" (UniqueName: \"kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z\") pod \"9af9a4ba-e1ee-4481-89de-acb671353972\" (UID: \"9af9a4ba-e1ee-4481-89de-acb671353972\") " Oct 31 01:45:22.077878 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.077831 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca" pod "9af9a4ba-e1ee-4481-89de-acb671353972" (UID: "9af9a4ba-e1ee-4481-89de-acb671353972"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:22.078035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.078018 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9a4ba-e1ee-4481-89de-acb671353972-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:22.079795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.079769 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z" pod "9af9a4ba-e1ee-4481-89de-acb671353972" (UID: "9af9a4ba-e1ee-4481-89de-acb671353972"). InnerVolumeSpecName "kube-api-access-lrc4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:22.179671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.179598 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrc4z\" (UniqueName: \"kubernetes.io/projected/9af9a4ba-e1ee-4481-89de-acb671353972-kube-api-access-lrc4z\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:22.310526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.310492 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" path="/var/lib/kubelet/pods/77c6093f-61cc-4b3c-97df-776555390fd7/volumes" Oct 31 01:45:22.310920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.310907 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" path="/var/lib/kubelet/pods/b53321bd-0aff-4ef5-8073-3a767cbd0329/volumes" Oct 31 01:45:22.311268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.311256 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" path="/var/lib/kubelet/pods/ebd19de2-2772-40fb-ab58-2e35f5ef3c2a/volumes" Oct 31 01:45:22.448004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.447925 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" Oct 31 01:45:22.448004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.447956 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5" event={"ID":"9af9a4ba-e1ee-4481-89de-acb671353972","Type":"ContainerDied","Data":"0c34c2d36c35cea3b7039cb4a6f90afcd6bb4110130974e9c974f1a46a660809"} Oct 31 01:45:22.448494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.448031 5756 scope.go:122] "RemoveContainer" containerID="98a040a8c7995fd8717d47f3a47ea8a36944e79690d7e9f7f0fc795e514e2c61" Oct 31 01:45:22.609928 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.609890 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5"] Oct 31 01:45:22.612573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:22.612546 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-001a26-29890185-8d9b5"] Oct 31 01:45:23.031598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.031399 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28"] Oct 31 01:45:23.031995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.031976 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.031995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.031994 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032005 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032010 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032025 5756 cpu_manager.go:414] "containerMap: removing container" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032030 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032039 5756 cpu_manager.go:414] "containerMap: removing container" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032043 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032056 5756 cpu_manager.go:414] "containerMap: removing container" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032061 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032071 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032075 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032082 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032086 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032095 5756 cpu_manager.go:414] "containerMap: removing container" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032101 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032107 5756 cpu_manager.go:414] "containerMap: removing container" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032114 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032120 5756 cpu_manager.go:414] "containerMap: removing container" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032124 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032133 5756 cpu_manager.go:414] "containerMap: removing container" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032137 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032143 5756 cpu_manager.go:414] "containerMap: removing container" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032148 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032157 5756 cpu_manager.go:414] "containerMap: removing container" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032161 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032168 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032173 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032181 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032185 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032192 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.032181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032196 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032208 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032213 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032221 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0b3c7885-9bc0-4daa-8381-0fb2f83b9831" containerName="revoke-keys" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032225 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3c7885-9bc0-4daa-8381-0fb2f83b9831" containerName="revoke-keys" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032236 5756 cpu_manager.go:414] "containerMap: removing container" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032239 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032305 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032312 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032318 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032324 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032329 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032334 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032338 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032344 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032348 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032353 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032360 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032365 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="377cfa2f-5923-4732-a87e-bedcc17090ba" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032371 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032378 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032382 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032387 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032392 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032397 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b544add2-4eaa-4fc5-aef6-80a1dd872786" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032403 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:23.033601 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.032409 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3c7885-9bc0-4daa-8381-0fb2f83b9831" containerName="revoke-keys" Oct 31 01:45:23.037504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.037485 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.089631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.089600 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.089794 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.089652 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6jv\" (UniqueName: \"kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.089913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.089884 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.090004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.089913 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:45:23.090121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.090053 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:45:23.092350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.092323 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28"] Oct 31 01:45:23.191046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.191012 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.191240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.191053 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6jv\" (UniqueName: \"kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.191240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.191143 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.191706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.191687 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.193592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.193568 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.248251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.248214 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6jv\" (UniqueName: \"kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv\") pod \"maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.356280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.356203 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:23.727549 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:23.727524 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28"] Oct 31 01:45:23.728969 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:23.728940 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59111aa7_c90b_4a00_9891_49e28a26397a.slice/crio-50c5a1aa84524aa0468465d98827f43b33c17238ecd33aeabd0bb8dc6be45ed8 WatchSource:0}: Error finding container 50c5a1aa84524aa0468465d98827f43b33c17238ecd33aeabd0bb8dc6be45ed8: Status 404 returned error can't find the container with id 50c5a1aa84524aa0468465d98827f43b33c17238ecd33aeabd0bb8dc6be45ed8 Oct 31 01:45:24.311048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:24.310971 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" path="/var/lib/kubelet/pods/9af9a4ba-e1ee-4481-89de-acb671353972/volumes" Oct 31 01:45:24.472031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:24.471997 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" event={"ID":"59111aa7-c90b-4a00-9891-49e28a26397a","Type":"ContainerStarted","Data":"9f15bf7e4db9dd02700d435d55aed9a5f5480239493859261938aeed5ab39ff0"} Oct 31 01:45:24.472203 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:24.472083 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" event={"ID":"59111aa7-c90b-4a00-9891-49e28a26397a","Type":"ContainerStarted","Data":"50c5a1aa84524aa0468465d98827f43b33c17238ecd33aeabd0bb8dc6be45ed8"} Oct 31 01:45:24.586987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:24.586894 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" podStartSLOduration=2.586849448 podStartE2EDuration="2.586849448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:24.585233098 +0000 UTC m=+2896.392200643" watchObservedRunningTime="2026-10-31 01:45:24.586849448 +0000 UTC m=+2896.393816990" Oct 31 01:45:25.480653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:25.480621 5756 generic.go:356] "Generic (PLEG): container finished" podID="59111aa7-c90b-4a00-9891-49e28a26397a" containerID="9f15bf7e4db9dd02700d435d55aed9a5f5480239493859261938aeed5ab39ff0" exitCode=0 Oct 31 01:45:25.481104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:25.480702 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" event={"ID":"59111aa7-c90b-4a00-9891-49e28a26397a","Type":"ContainerDied","Data":"9f15bf7e4db9dd02700d435d55aed9a5f5480239493859261938aeed5ab39ff0"} Oct 31 01:45:26.639512 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.639485 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:26.730042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.730016 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca\") pod \"59111aa7-c90b-4a00-9891-49e28a26397a\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " Oct 31 01:45:26.730708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.730176 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca\") pod \"59111aa7-c90b-4a00-9891-49e28a26397a\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " Oct 31 01:45:26.730708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.730315 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv\" (UniqueName: \"kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv\") pod \"59111aa7-c90b-4a00-9891-49e28a26397a\" (UID: \"59111aa7-c90b-4a00-9891-49e28a26397a\") " Oct 31 01:45:26.730708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.730484 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca" pod "59111aa7-c90b-4a00-9891-49e28a26397a" (UID: "59111aa7-c90b-4a00-9891-49e28a26397a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:26.730923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.730723 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:26.732508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.732479 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca" pod "59111aa7-c90b-4a00-9891-49e28a26397a" (UID: "59111aa7-c90b-4a00-9891-49e28a26397a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:45:26.732622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.732527 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv" pod "59111aa7-c90b-4a00-9891-49e28a26397a" (UID: "59111aa7-c90b-4a00-9891-49e28a26397a"). InnerVolumeSpecName "kube-api-access-cm6jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:26.831938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.831905 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59111aa7-c90b-4a00-9891-49e28a26397a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:26.831938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:26.831932 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm6jv\" (UniqueName: \"kubernetes.io/projected/59111aa7-c90b-4a00-9891-49e28a26397a-kube-api-access-cm6jv\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:27.496482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:27.496449 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" Oct 31 01:45:27.496482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:27.496461 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28" event={"ID":"59111aa7-c90b-4a00-9891-49e28a26397a","Type":"ContainerDied","Data":"50c5a1aa84524aa0468465d98827f43b33c17238ecd33aeabd0bb8dc6be45ed8"} Oct 31 01:45:27.496760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:27.496526 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50c5a1aa84524aa0468465d98827f43b33c17238ecd33aeabd0bb8dc6be45ed8" Oct 31 01:45:29.961477 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.961443 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2"] Oct 31 01:45:29.961993 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.961975 5756 cpu_manager.go:414] "containerMap: removing container" podUID="59111aa7-c90b-4a00-9891-49e28a26397a" containerName="revoke-keys" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.961995 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59111aa7-c90b-4a00-9891-49e28a26397a" containerName="revoke-keys" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962008 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962013 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd19de2-2772-40fb-ab58-2e35f5ef3c2a" containerName="cleanup" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962028 5756 cpu_manager.go:414] "containerMap: removing container" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962033 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c6093f-61cc-4b3c-97df-776555390fd7" containerName="cleanup" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962041 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:29.962046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962046 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:29.962262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962066 5756 cpu_manager.go:414] "containerMap: removing container" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:29.962262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962071 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c558a3-547a-4d2d-ac43-a19be6e4493d" containerName="cleanup" Oct 31 01:45:29.962262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962153 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7060ce-e966-4579-aa1d-964bef4a3efe" containerName="cleanup" Oct 31 01:45:29.962262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962161 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af9a4ba-e1ee-4481-89de-acb671353972" containerName="cleanup" Oct 31 01:45:29.962262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962173 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b53321bd-0aff-4ef5-8073-3a767cbd0329" containerName="cleanup" Oct 31 01:45:29.962262 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.962182 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="59111aa7-c90b-4a00-9891-49e28a26397a" containerName="revoke-keys" Oct 31 01:45:29.965697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:29.965682 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.066089 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.066052 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.066258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.066118 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbzd\" (UniqueName: \"kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.066308 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.066288 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.067035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.067018 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:45:30.067035 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.067027 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:45:30.069664 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.069644 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2"] Oct 31 01:45:30.167379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.167346 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.167600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.167413 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbzd\" (UniqueName: \"kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.167600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.167500 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.168128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.168102 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.170040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.170017 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.225011 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.224950 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbzd\" (UniqueName: \"kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd\") pod \"maas-api-revoke-sub-79452fa3-xwxl2\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.282205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.282177 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:30.658426 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:30.658395 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2"] Oct 31 01:45:30.659569 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:30.659536 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabf28dd_c287_46bd_9cb3_18ab9c564221.slice/crio-d9bce4431bf11f77e1b5b89c8a5cbc97329bbd4c29e2abaaa1812413ed5eb325 WatchSource:0}: Error finding container d9bce4431bf11f77e1b5b89c8a5cbc97329bbd4c29e2abaaa1812413ed5eb325: Status 404 returned error can't find the container with id d9bce4431bf11f77e1b5b89c8a5cbc97329bbd4c29e2abaaa1812413ed5eb325 Oct 31 01:45:31.524600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:31.524568 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" event={"ID":"eabf28dd-c287-46bd-9cb3-18ab9c564221","Type":"ContainerStarted","Data":"beea736867df7b4e8a1d647a2dd0f20efe4df7fd8383de2c83741f3a3a641b0d"} Oct 31 01:45:31.525087 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:31.524617 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" event={"ID":"eabf28dd-c287-46bd-9cb3-18ab9c564221","Type":"ContainerStarted","Data":"d9bce4431bf11f77e1b5b89c8a5cbc97329bbd4c29e2abaaa1812413ed5eb325"} Oct 31 01:45:31.640428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:31.640380 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" podStartSLOduration=2.640361638 podStartE2EDuration="2.640361638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:31.637615782 +0000 UTC m=+2903.444583326" watchObservedRunningTime="2026-10-31 01:45:31.640361638 +0000 UTC m=+2903.447329183" Oct 31 01:45:32.066364 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:32.066330 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:45:32.532704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:32.532673 5756 generic.go:356] "Generic (PLEG): container finished" podID="eabf28dd-c287-46bd-9cb3-18ab9c564221" containerID="beea736867df7b4e8a1d647a2dd0f20efe4df7fd8383de2c83741f3a3a641b0d" exitCode=0 Oct 31 01:45:32.533105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:32.532755 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" event={"ID":"eabf28dd-c287-46bd-9cb3-18ab9c564221","Type":"ContainerDied","Data":"beea736867df7b4e8a1d647a2dd0f20efe4df7fd8383de2c83741f3a3a641b0d"} Oct 31 01:45:33.506916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.506852 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6"] Oct 31 01:45:33.510821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.510805 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.610377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.609295 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.610377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.609396 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.610377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.609433 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz49p\" (UniqueName: \"kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.615696 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.615666 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6"] Oct 31 01:45:33.689885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.689836 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:33.710700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.710668 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.710892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.710712 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz49p\" (UniqueName: \"kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.710892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.710840 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.711355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.711336 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.713428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.713404 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.767627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.767552 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz49p\" (UniqueName: \"kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p\") pod \"maas-api-revoke-sub-a7e4fb4d-v2gh6\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.812244 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.812218 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca\") pod \"eabf28dd-c287-46bd-9cb3-18ab9c564221\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " Oct 31 01:45:33.812438 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.812285 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca\") pod \"eabf28dd-c287-46bd-9cb3-18ab9c564221\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " Oct 31 01:45:33.812438 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.812323 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd\" (UniqueName: \"kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd\") pod \"eabf28dd-c287-46bd-9cb3-18ab9c564221\" (UID: \"eabf28dd-c287-46bd-9cb3-18ab9c564221\") " Oct 31 01:45:33.812746 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.812718 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca" pod "eabf28dd-c287-46bd-9cb3-18ab9c564221" (UID: "eabf28dd-c287-46bd-9cb3-18ab9c564221"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:33.814577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.814545 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca" pod "eabf28dd-c287-46bd-9cb3-18ab9c564221" (UID: "eabf28dd-c287-46bd-9cb3-18ab9c564221"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:45:33.814577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.814552 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd" pod "eabf28dd-c287-46bd-9cb3-18ab9c564221" (UID: "eabf28dd-c287-46bd-9cb3-18ab9c564221"). InnerVolumeSpecName "kube-api-access-xkbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:33.823774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.823748 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:33.913739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.913702 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:33.913739 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.913736 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eabf28dd-c287-46bd-9cb3-18ab9c564221-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:33.913958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:33.913753 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbzd\" (UniqueName: \"kubernetes.io/projected/eabf28dd-c287-46bd-9cb3-18ab9c564221-kube-api-access-xkbzd\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:34.197029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.197000 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6"] Oct 31 01:45:34.198574 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:34.198546 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9fb9eb3_c456_43ff_be56_186b3469dd4d.slice/crio-154e36db6aa5ee9c3cf80961a09b72ad6a33a9cc5674b83ae2f5cee71833e3d2 WatchSource:0}: Error finding container 154e36db6aa5ee9c3cf80961a09b72ad6a33a9cc5674b83ae2f5cee71833e3d2: Status 404 returned error can't find the container with id 154e36db6aa5ee9c3cf80961a09b72ad6a33a9cc5674b83ae2f5cee71833e3d2 Oct 31 01:45:34.502029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.501941 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9"] Oct 31 01:45:34.548552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.548530 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" Oct 31 01:45:34.548701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.548566 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2" event={"ID":"eabf28dd-c287-46bd-9cb3-18ab9c564221","Type":"ContainerDied","Data":"d9bce4431bf11f77e1b5b89c8a5cbc97329bbd4c29e2abaaa1812413ed5eb325"} Oct 31 01:45:34.548701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.548621 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9bce4431bf11f77e1b5b89c8a5cbc97329bbd4c29e2abaaa1812413ed5eb325" Oct 31 01:45:34.550715 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.550655 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" event={"ID":"b9fb9eb3-c456-43ff-be56-186b3469dd4d","Type":"ContainerStarted","Data":"a1b87ab0d8350c7ad594b33431b6838929e59266ca4fb51345f103044264d11e"} Oct 31 01:45:34.550854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.550745 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" event={"ID":"b9fb9eb3-c456-43ff-be56-186b3469dd4d","Type":"ContainerStarted","Data":"154e36db6aa5ee9c3cf80961a09b72ad6a33a9cc5674b83ae2f5cee71833e3d2"} Oct 31 01:45:34.606510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.606481 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd"] Oct 31 01:45:34.888594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:34.888543 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" podStartSLOduration=1.888529552 podStartE2EDuration="1.888529552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:34.88547263 +0000 UTC m=+2906.692440176" watchObservedRunningTime="2026-10-31 01:45:34.888529552 +0000 UTC m=+2906.695497094" Oct 31 01:45:35.558111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:35.558075 5756 generic.go:356] "Generic (PLEG): container finished" podID="b9fb9eb3-c456-43ff-be56-186b3469dd4d" containerID="a1b87ab0d8350c7ad594b33431b6838929e59266ca4fb51345f103044264d11e" exitCode=0 Oct 31 01:45:35.558299 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:35.558163 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" event={"ID":"b9fb9eb3-c456-43ff-be56-186b3469dd4d","Type":"ContainerDied","Data":"a1b87ab0d8350c7ad594b33431b6838929e59266ca4fb51345f103044264d11e"} Oct 31 01:45:36.043475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.043431 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l"] Oct 31 01:45:36.047471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.047441 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f758a2a6-th42l"] Oct 31 01:45:36.314157 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.314036 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35dc0018-4a87-42ce-850d-91e328b9d54b" path="/var/lib/kubelet/pods/35dc0018-4a87-42ce-850d-91e328b9d54b/volumes" Oct 31 01:45:36.715569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.715544 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:36.851270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.851180 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p\" (UniqueName: \"kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p\") pod \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " Oct 31 01:45:36.851270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.851242 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca\") pod \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " Oct 31 01:45:36.851505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.851275 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca\") pod \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\" (UID: \"b9fb9eb3-c456-43ff-be56-186b3469dd4d\") " Oct 31 01:45:36.851693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.851660 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca" pod "b9fb9eb3-c456-43ff-be56-186b3469dd4d" (UID: "b9fb9eb3-c456-43ff-be56-186b3469dd4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:36.851826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.851799 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:36.856038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.856013 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca" pod "b9fb9eb3-c456-43ff-be56-186b3469dd4d" (UID: "b9fb9eb3-c456-43ff-be56-186b3469dd4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:45:36.858036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.858006 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p" pod "b9fb9eb3-c456-43ff-be56-186b3469dd4d" (UID: "b9fb9eb3-c456-43ff-be56-186b3469dd4d"). InnerVolumeSpecName "kube-api-access-xz49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:36.953067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.953034 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz49p\" (UniqueName: \"kubernetes.io/projected/b9fb9eb3-c456-43ff-be56-186b3469dd4d-kube-api-access-xz49p\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:36.953067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:36.953062 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9fb9eb3-c456-43ff-be56-186b3469dd4d-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:37.573682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:37.573648 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" event={"ID":"b9fb9eb3-c456-43ff-be56-186b3469dd4d","Type":"ContainerDied","Data":"154e36db6aa5ee9c3cf80961a09b72ad6a33a9cc5674b83ae2f5cee71833e3d2"} Oct 31 01:45:37.573682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:37.573671 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6" Oct 31 01:45:37.574172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:37.573698 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154e36db6aa5ee9c3cf80961a09b72ad6a33a9cc5674b83ae2f5cee71833e3d2" Oct 31 01:45:38.013695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.013616 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th"] Oct 31 01:45:38.014163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.014150 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b9fb9eb3-c456-43ff-be56-186b3469dd4d" containerName="revoke-keys" Oct 31 01:45:38.014217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.014165 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9fb9eb3-c456-43ff-be56-186b3469dd4d" containerName="revoke-keys" Oct 31 01:45:38.014217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.014183 5756 cpu_manager.go:414] "containerMap: removing container" podUID="eabf28dd-c287-46bd-9cb3-18ab9c564221" containerName="revoke-keys" Oct 31 01:45:38.014217 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.014188 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabf28dd-c287-46bd-9cb3-18ab9c564221" containerName="revoke-keys" Oct 31 01:45:38.014311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.014258 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9fb9eb3-c456-43ff-be56-186b3469dd4d" containerName="revoke-keys" Oct 31 01:45:38.014311 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.014268 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabf28dd-c287-46bd-9cb3-18ab9c564221" containerName="revoke-keys" Oct 31 01:45:38.016502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.016487 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.119040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.119010 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th"] Oct 31 01:45:38.166318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166286 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166328 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166424 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnbc\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166545 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166569 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166665 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166716 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166709 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166791 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166780 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.166829 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.166810 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268303 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268353 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268397 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnbc\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268447 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268471 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268539 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268598 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268666 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268706 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268814 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.268921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.268913 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.269143 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.269088 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.269200 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.269182 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.269303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.269282 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.270807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.270774 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.271397 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.271380 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.325738 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.325699 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.325929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.325776 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnbc\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc\") pod \"e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.329538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.329518 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:38.757766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.757738 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th"] Oct 31 01:45:38.758802 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:38.758774 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d47ab4c_92ab_4632_a924_aaf53a48de97.slice/crio-d374e9756e3cb27761b75f753cf6ada834af353f3edb27c23182fcbc92f8ad95 WatchSource:0}: Error finding container d374e9756e3cb27761b75f753cf6ada834af353f3edb27c23182fcbc92f8ad95: Status 404 returned error can't find the container with id d374e9756e3cb27761b75f753cf6ada834af353f3edb27c23182fcbc92f8ad95 Oct 31 01:45:38.761336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.761302 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:45:38.761416 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.761376 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:45:38.761416 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:38.761407 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:45:39.503331 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.503268 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" podUID="db0908f8-6f62-468b-9829-f542f699a6c6" containerName="payload-processing" containerID="cri-o://b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f" gracePeriod=25 Oct 31 01:45:39.589575 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.589533 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" event={"ID":"8d47ab4c-92ab-4632-a924-aaf53a48de97","Type":"ContainerStarted","Data":"122e9d2e704730873717caec3d67d6b83cc3fa1d7a2f4f89b3a1c712598d01ea"} Oct 31 01:45:39.589761 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.589600 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" event={"ID":"8d47ab4c-92ab-4632-a924-aaf53a48de97","Type":"ContainerStarted","Data":"d374e9756e3cb27761b75f753cf6ada834af353f3edb27c23182fcbc92f8ad95"} Oct 31 01:45:39.606767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.606729 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" podUID="66905eaa-938e-415e-a241-0e29d0ce498a" containerName="payload-pre-processing" containerID="cri-o://27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851" gracePeriod=25 Oct 31 01:45:39.712679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.712597 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podStartSLOduration=2.712577882 podStartE2EDuration="2.712577882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:39.707365669 +0000 UTC m=+2911.514333225" watchObservedRunningTime="2026-10-31 01:45:39.712577882 +0000 UTC m=+2911.519545427" Oct 31 01:45:39.816504 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.816477 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:45:39.912039 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.912013 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:45:39.990138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990106 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl\" (UniqueName: \"kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl\") pod \"db0908f8-6f62-468b-9829-f542f699a6c6\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " Oct 31 01:45:39.990322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990182 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume\") pod \"66905eaa-938e-415e-a241-0e29d0ce498a\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " Oct 31 01:45:39.990322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990281 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr\" (UniqueName: \"kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr\") pod \"66905eaa-938e-415e-a241-0e29d0ce498a\" (UID: \"66905eaa-938e-415e-a241-0e29d0ce498a\") " Oct 31 01:45:39.990399 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990334 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume\") pod \"db0908f8-6f62-468b-9829-f542f699a6c6\" (UID: \"db0908f8-6f62-468b-9829-f542f699a6c6\") " Oct 31 01:45:39.990662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990564 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume" pod "66905eaa-938e-415e-a241-0e29d0ce498a" (UID: "66905eaa-938e-415e-a241-0e29d0ce498a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:39.990813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990735 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume" pod "db0908f8-6f62-468b-9829-f542f699a6c6" (UID: "db0908f8-6f62-468b-9829-f542f699a6c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:39.990813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.990759 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66905eaa-938e-415e-a241-0e29d0ce498a-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:39.992485 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.992466 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr" pod "66905eaa-938e-415e-a241-0e29d0ce498a" (UID: "66905eaa-938e-415e-a241-0e29d0ce498a"). InnerVolumeSpecName "kube-api-access-9x5pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:39.992579 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:39.992495 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl" pod "db0908f8-6f62-468b-9829-f542f699a6c6" (UID: "db0908f8-6f62-468b-9829-f542f699a6c6"). InnerVolumeSpecName "kube-api-access-6tncl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:40.092393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.092302 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x5pr\" (UniqueName: \"kubernetes.io/projected/66905eaa-938e-415e-a241-0e29d0ce498a-kube-api-access-9x5pr\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:40.092393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.092332 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db0908f8-6f62-468b-9829-f542f699a6c6-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:40.092393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.092344 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tncl\" (UniqueName: \"kubernetes.io/projected/db0908f8-6f62-468b-9829-f542f699a6c6-kube-api-access-6tncl\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:40.350961 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.335472 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:40.350961 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.335590 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:45:40.350961 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.335634 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:45:40.596996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.596955 5756 generic.go:356] "Generic (PLEG): container finished" podID="66905eaa-938e-415e-a241-0e29d0ce498a" containerID="27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851" exitCode=0 Oct 31 01:45:40.596996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.597000 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" Oct 31 01:45:40.597261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.596994 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" event={"ID":"66905eaa-938e-415e-a241-0e29d0ce498a","Type":"ContainerDied","Data":"27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851"} Oct 31 01:45:40.597261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.597133 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd" event={"ID":"66905eaa-938e-415e-a241-0e29d0ce498a","Type":"ContainerDied","Data":"43009ed8af0470dfd972d5356bff744b37ed629c300886aa96e2c40b330da14a"} Oct 31 01:45:40.597261 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.597164 5756 scope.go:122] "RemoveContainer" containerID="27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851" Oct 31 01:45:40.598923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.598900 5756 generic.go:356] "Generic (PLEG): container finished" podID="db0908f8-6f62-468b-9829-f542f699a6c6" containerID="b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f" exitCode=0 Oct 31 01:45:40.599042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.598945 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" Oct 31 01:45:40.599042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.598982 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" event={"ID":"db0908f8-6f62-468b-9829-f542f699a6c6","Type":"ContainerDied","Data":"b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f"} Oct 31 01:45:40.599042 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.599024 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9" event={"ID":"db0908f8-6f62-468b-9829-f542f699a6c6","Type":"ContainerDied","Data":"b4ccc762765ef8f9b9ba244cedf60788794050783709008bec561d331e1c3380"} Oct 31 01:45:40.607829 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.607811 5756 scope.go:122] "RemoveContainer" containerID="27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851" Oct 31 01:45:40.608126 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:40.608105 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851\": container with ID starting with 27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851 not found: ID does not exist" containerID="27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851" Oct 31 01:45:40.608176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.608135 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851"} err="failed to get container status \"27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851\": rpc error: code = NotFound desc = could not find container \"27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851\": container with ID starting with 27f8055fa7df89b94a46be4e50277df2eda2aa0322767ffbb2e04c6fe2e17851 not found: ID does not exist" Oct 31 01:45:40.608176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.608151 5756 scope.go:122] "RemoveContainer" containerID="b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f" Oct 31 01:45:40.616973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.616950 5756 scope.go:122] "RemoveContainer" containerID="b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f" Oct 31 01:45:40.617242 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:45:40.617225 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f\": container with ID starting with b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f not found: ID does not exist" containerID="b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f" Oct 31 01:45:40.617298 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.617248 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f"} err="failed to get container status \"b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f\": rpc error: code = NotFound desc = could not find container \"b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f\": container with ID starting with b26fb5f65da3381be34bbfcc286c4f7f59ef29a59f2fd328f828519b49aeb17f not found: ID does not exist" Oct 31 01:45:40.767018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.766985 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9"] Oct 31 01:45:40.769479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.769446 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-0c0723-757b66d6df-tb7m9"] Oct 31 01:45:40.876334 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.876253 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd"] Oct 31 01:45:40.880527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:40.880500 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0c0723-7b8d4659-ndnnd"] Oct 31 01:45:41.353256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:41.353219 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:45:41.353459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:41.353283 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:45:41.554983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:41.549723 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:45:42.312303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:42.312265 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66905eaa-938e-415e-a241-0e29d0ce498a" path="/var/lib/kubelet/pods/66905eaa-938e-415e-a241-0e29d0ce498a/volumes" Oct 31 01:45:42.312821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:42.312804 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0908f8-6f62-468b-9829-f542f699a6c6" path="/var/lib/kubelet/pods/db0908f8-6f62-468b-9829-f542f699a6c6/volumes" Oct 31 01:45:42.332301 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:42.332265 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:45:42.332576 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:42.332551 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:45:43.346889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:43.337226 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:45:43.346889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:43.337298 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:45:44.349185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:44.349149 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:45:44.349759 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:44.349728 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:45:45.330667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.330630 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 31 01:45:45.330847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.330693 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 31 01:45:45.595891 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.595802 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp"] Oct 31 01:45:45.596597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.596576 5756 cpu_manager.go:414] "containerMap: removing container" podUID="66905eaa-938e-415e-a241-0e29d0ce498a" containerName="payload-pre-processing" Oct 31 01:45:45.596651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.596602 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66905eaa-938e-415e-a241-0e29d0ce498a" containerName="payload-pre-processing" Oct 31 01:45:45.596651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.596625 5756 cpu_manager.go:414] "containerMap: removing container" podUID="db0908f8-6f62-468b-9829-f542f699a6c6" containerName="payload-processing" Oct 31 01:45:45.596651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.596636 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0908f8-6f62-468b-9829-f542f699a6c6" containerName="payload-processing" Oct 31 01:45:45.596797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.596783 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="66905eaa-938e-415e-a241-0e29d0ce498a" containerName="payload-pre-processing" Oct 31 01:45:45.596832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.596803 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0908f8-6f62-468b-9829-f542f699a6c6" containerName="payload-processing" Oct 31 01:45:45.600695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.600673 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.655349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.655317 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp"] Oct 31 01:45:45.683366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.683333 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.683575 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.683395 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.683657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.683565 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvzq\" (UniqueName: \"kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.700059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.700017 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:45:45.700213 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.700081 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:45:45.785685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.785652 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.785909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.785712 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.785909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.785793 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvzq\" (UniqueName: \"kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.786742 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.786714 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.788523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.788493 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.842167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.842131 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvzq\" (UniqueName: \"kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq\") pod \"maas-api-revoke-sub-3f771f4c-xmdwp\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:45.913629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:45.913547 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:46.291082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:46.291057 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp"] Oct 31 01:45:46.294069 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:45:46.294038 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648064f7_8cbb_4231_a83c_e68ac986dd80.slice/crio-3fec1faac564615e3b7d6504a22782daf18b631c8c0263ad77f67e19c725a9d7 WatchSource:0}: Error finding container 3fec1faac564615e3b7d6504a22782daf18b631c8c0263ad77f67e19c725a9d7: Status 404 returned error can't find the container with id 3fec1faac564615e3b7d6504a22782daf18b631c8c0263ad77f67e19c725a9d7 Oct 31 01:45:46.652014 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:46.651983 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" event={"ID":"648064f7-8cbb-4231-a83c-e68ac986dd80","Type":"ContainerStarted","Data":"81201cc6d5b31879503021568dd53958fb05bdf0ec86fbe41cdeeb0ee840abed"} Oct 31 01:45:46.652421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:46.652043 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" event={"ID":"648064f7-8cbb-4231-a83c-e68ac986dd80","Type":"ContainerStarted","Data":"3fec1faac564615e3b7d6504a22782daf18b631c8c0263ad77f67e19c725a9d7"} Oct 31 01:45:46.766516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:46.766456 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" podStartSLOduration=1.7664430260000001 podStartE2EDuration="1.766443026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:46.763793579 +0000 UTC m=+2918.570761128" watchObservedRunningTime="2026-10-31 01:45:46.766443026 +0000 UTC m=+2918.573410569" Oct 31 01:45:47.372032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.371983 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:45:47.372321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.372068 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:45:47.566194 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.566114 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:47.567300 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.567275 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:47.568273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.568256 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:45:47.661002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.660963 5756 generic.go:356] "Generic (PLEG): container finished" podID="648064f7-8cbb-4231-a83c-e68ac986dd80" containerID="81201cc6d5b31879503021568dd53958fb05bdf0ec86fbe41cdeeb0ee840abed" exitCode=0 Oct 31 01:45:47.661406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:47.661048 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" event={"ID":"648064f7-8cbb-4231-a83c-e68ac986dd80","Type":"ContainerDied","Data":"81201cc6d5b31879503021568dd53958fb05bdf0ec86fbe41cdeeb0ee840abed"} Oct 31 01:45:48.816278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.816253 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:48.927476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.927391 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca\") pod \"648064f7-8cbb-4231-a83c-e68ac986dd80\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " Oct 31 01:45:48.927645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.927523 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq\" (UniqueName: \"kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq\") pod \"648064f7-8cbb-4231-a83c-e68ac986dd80\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " Oct 31 01:45:48.927645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.927570 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca\") pod \"648064f7-8cbb-4231-a83c-e68ac986dd80\" (UID: \"648064f7-8cbb-4231-a83c-e68ac986dd80\") " Oct 31 01:45:48.928046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.928011 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca" pod "648064f7-8cbb-4231-a83c-e68ac986dd80" (UID: "648064f7-8cbb-4231-a83c-e68ac986dd80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:45:48.930190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.930164 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq" pod "648064f7-8cbb-4231-a83c-e68ac986dd80" (UID: "648064f7-8cbb-4231-a83c-e68ac986dd80"). InnerVolumeSpecName "kube-api-access-ktvzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:45:48.930538 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:48.930513 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca" pod "648064f7-8cbb-4231-a83c-e68ac986dd80" (UID: "648064f7-8cbb-4231-a83c-e68ac986dd80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:45:49.029222 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:49.029192 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:49.029222 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:49.029221 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvzq\" (UniqueName: \"kubernetes.io/projected/648064f7-8cbb-4231-a83c-e68ac986dd80-kube-api-access-ktvzq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:49.029409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:49.029232 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648064f7-8cbb-4231-a83c-e68ac986dd80-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:45:49.675597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:49.675569 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" Oct 31 01:45:49.675801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:49.675596 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp" event={"ID":"648064f7-8cbb-4231-a83c-e68ac986dd80","Type":"ContainerDied","Data":"3fec1faac564615e3b7d6504a22782daf18b631c8c0263ad77f67e19c725a9d7"} Oct 31 01:45:49.675801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:45:49.675652 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fec1faac564615e3b7d6504a22782daf18b631c8c0263ad77f67e19c725a9d7" Oct 31 01:46:02.066798 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:46:02.066760 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:46:06.507313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:06.507283 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm"] Oct 31 01:46:06.524732 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:06.524700 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-66100b-tcxgm"] Oct 31 01:46:08.200167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.200133 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn"] Oct 31 01:46:08.200639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.200625 5756 cpu_manager.go:414] "containerMap: removing container" podUID="648064f7-8cbb-4231-a83c-e68ac986dd80" containerName="revoke-keys" Oct 31 01:46:08.200689 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.200641 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648064f7-8cbb-4231-a83c-e68ac986dd80" containerName="revoke-keys" Oct 31 01:46:08.200779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.200770 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="648064f7-8cbb-4231-a83c-e68ac986dd80" containerName="revoke-keys" Oct 31 01:46:08.204216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.204196 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.312274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.312242 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6185b5-5698-4d84-9999-b97ddc33873f" path="/var/lib/kubelet/pods/cc6185b5-5698-4d84-9999-b97ddc33873f/volumes" Oct 31 01:46:08.312556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.312541 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn"] Oct 31 01:46:08.341237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341200 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341246 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341371 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341410 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341429 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341528 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqh5\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341562 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341659 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.341753 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.341739 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.442932 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.442904 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.442998 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443022 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443057 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443075 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443123 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443092 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443411 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443136 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqh5\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443411 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443164 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443411 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443213 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443453 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443678 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443655 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443742 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.443834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.443797 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.444093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.444057 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.445802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.445780 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.446062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.446045 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.499730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.499660 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.500196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.500172 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqh5\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.518067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.518044 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:08.948118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.948076 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn"] Oct 31 01:46:08.949001 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:08.948973 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924bc7ad_57f1_4d79_a666_ce618a60ca4c.slice/crio-43e82d61cfb8f3001631a6e9f810644c0a18b6765442d807858d512049e69d1f WatchSource:0}: Error finding container 43e82d61cfb8f3001631a6e9f810644c0a18b6765442d807858d512049e69d1f: Status 404 returned error can't find the container with id 43e82d61cfb8f3001631a6e9f810644c0a18b6765442d807858d512049e69d1f Oct 31 01:46:08.951667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.951627 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:46:08.951765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.951712 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:46:08.951765 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:08.951754 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:46:09.817284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:09.817251 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" event={"ID":"924bc7ad-57f1-4d79-a666-ce618a60ca4c","Type":"ContainerStarted","Data":"65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a"} Oct 31 01:46:09.817802 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:09.817306 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" event={"ID":"924bc7ad-57f1-4d79-a666-ce618a60ca4c","Type":"ContainerStarted","Data":"43e82d61cfb8f3001631a6e9f810644c0a18b6765442d807858d512049e69d1f"} Oct 31 01:46:10.162414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:10.162328 5756 scope.go:122] "RemoveContainer" containerID="a4d5658d613f847c2c7fa61db169597007d365f2c35f5ca2756e8942520e3e1a" Oct 31 01:46:10.172208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:10.172175 5756 scope.go:122] "RemoveContainer" containerID="9c79d8ef18dfbb46c0d70e57d2036e7980321762a917c94e71ebf64db385d6ab" Oct 31 01:46:10.184783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:10.184763 5756 scope.go:122] "RemoveContainer" containerID="50c4bd97a0ccdc026d8b976c243c8264d6a868d0e107ea8c899bf9dc113444e1" Oct 31 01:46:10.519169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:10.519086 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:10.520777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:10.520751 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:46:10.520945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:10.520805 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:46:11.518650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:11.518605 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:46:11.519058 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:11.518670 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:46:12.519141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:12.519105 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:46:12.519554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:12.519157 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:46:13.518494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:13.518454 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:46:13.518705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:13.518511 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:46:14.157407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.157355 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podStartSLOduration=6.157340313 podStartE2EDuration="6.157340313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:10.035524258 +0000 UTC m=+2941.842491837" watchObservedRunningTime="2026-10-31 01:46:14.157340313 +0000 UTC m=+2945.964307850" Oct 31 01:46:14.158607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.158583 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d"] Oct 31 01:46:14.164576 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.164558 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.211295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.211269 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.211443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.211323 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.211443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.211387 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsffq\" (UniqueName: \"kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.215830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.215806 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d"] Oct 31 01:46:14.216392 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.216374 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:46:14.216471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.216407 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:46:14.312615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.312582 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.312784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.312638 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.312784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.312664 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsffq\" (UniqueName: \"kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.313223 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.313195 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.315236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.315218 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.370333 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.370299 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsffq\" (UniqueName: \"kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq\") pod \"maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.479032 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.478951 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:14.519124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.519089 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:46:14.519271 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.519150 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:46:14.907144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:14.907120 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d"] Oct 31 01:46:14.909074 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:14.909044 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf84559_aed8_4bbf_8770_296712aefb9e.slice/crio-db5100412d1df9310f04e3997778a5c3240caadd4be1940e1b6a5401c9d443d7 WatchSource:0}: Error finding container db5100412d1df9310f04e3997778a5c3240caadd4be1940e1b6a5401c9d443d7: Status 404 returned error can't find the container with id db5100412d1df9310f04e3997778a5c3240caadd4be1940e1b6a5401c9d443d7 Oct 31 01:46:15.518997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.518912 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 31 01:46:15.518997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.518964 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 31 01:46:15.739458 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.739428 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp"] Oct 31 01:46:15.751474 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.751445 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.802002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.801854 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp"] Oct 31 01:46:15.827811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.827773 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.828019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.827984 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.828097 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.828072 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2h2l\" (UniqueName: \"kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.885755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.885708 5756 generic.go:356] "Generic (PLEG): container finished" podID="fdf84559-aed8-4bbf-8770-296712aefb9e" containerID="fbb970336af33b98484cc88caf8bea7e309ce02e20b1d9ec4e87b303bc857e48" exitCode=0 Oct 31 01:46:15.885945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.885757 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" event={"ID":"fdf84559-aed8-4bbf-8770-296712aefb9e","Type":"ContainerDied","Data":"fbb970336af33b98484cc88caf8bea7e309ce02e20b1d9ec4e87b303bc857e48"} Oct 31 01:46:15.885945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.885806 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" event={"ID":"fdf84559-aed8-4bbf-8770-296712aefb9e","Type":"ContainerStarted","Data":"db5100412d1df9310f04e3997778a5c3240caadd4be1940e1b6a5401c9d443d7"} Oct 31 01:46:15.929461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.929429 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.929630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.929508 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.929630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.929565 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2h2l\" (UniqueName: \"kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.930417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.930386 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.932006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.931986 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:15.986303 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:15.986276 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2h2l\" (UniqueName: \"kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l\") pod \"maas-api-revoke-sub-9295b6c5-8gcbp\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:16.065960 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.065858 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:16.357856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.357785 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:46:16.466220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.466197 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp"] Oct 31 01:46:16.468033 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:16.467998 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6993e5_4f4b_42e8_b119_19c1c88bc46e.slice/crio-e5a95f74b9d404dbd40ad848d1cafbe03161aa1870963cded3620d627c542f0d WatchSource:0}: Error finding container e5a95f74b9d404dbd40ad848d1cafbe03161aa1870963cded3620d627c542f0d: Status 404 returned error can't find the container with id e5a95f74b9d404dbd40ad848d1cafbe03161aa1870963cded3620d627c542f0d Oct 31 01:46:16.522993 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.522970 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:16.523410 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.523246 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:16.524105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.524085 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:46:16.893935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.893831 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" event={"ID":"fd6993e5-4f4b-42e8-b119-19c1c88bc46e","Type":"ContainerStarted","Data":"2380fb24bc4cba58915ebc08c321574dd6f15eccc0b305c021e8e9402088c771"} Oct 31 01:46:16.893935 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:16.893926 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" event={"ID":"fd6993e5-4f4b-42e8-b119-19c1c88bc46e","Type":"ContainerStarted","Data":"e5a95f74b9d404dbd40ad848d1cafbe03161aa1870963cded3620d627c542f0d"} Oct 31 01:46:17.010001 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.009945 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" podStartSLOduration=2.009927884 podStartE2EDuration="2.009927884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:17.009069006 +0000 UTC m=+2948.816036550" watchObservedRunningTime="2026-10-31 01:46:17.009927884 +0000 UTC m=+2948.816895455" Oct 31 01:46:17.065609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.065582 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:17.150151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.150053 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca\") pod \"fdf84559-aed8-4bbf-8770-296712aefb9e\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " Oct 31 01:46:17.150315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.150279 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca\") pod \"fdf84559-aed8-4bbf-8770-296712aefb9e\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " Oct 31 01:46:17.150383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.150323 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq\" (UniqueName: \"kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq\") pod \"fdf84559-aed8-4bbf-8770-296712aefb9e\" (UID: \"fdf84559-aed8-4bbf-8770-296712aefb9e\") " Oct 31 01:46:17.150455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.150421 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca" pod "fdf84559-aed8-4bbf-8770-296712aefb9e" (UID: "fdf84559-aed8-4bbf-8770-296712aefb9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:17.150810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.150777 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:17.152633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.152605 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca" pod "fdf84559-aed8-4bbf-8770-296712aefb9e" (UID: "fdf84559-aed8-4bbf-8770-296712aefb9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:17.152633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.152607 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq" pod "fdf84559-aed8-4bbf-8770-296712aefb9e" (UID: "fdf84559-aed8-4bbf-8770-296712aefb9e"). InnerVolumeSpecName "kube-api-access-wsffq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:17.251912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.251856 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf84559-aed8-4bbf-8770-296712aefb9e-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:17.251912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.251916 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsffq\" (UniqueName: \"kubernetes.io/projected/fdf84559-aed8-4bbf-8770-296712aefb9e-kube-api-access-wsffq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:17.912359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.912320 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" event={"ID":"fdf84559-aed8-4bbf-8770-296712aefb9e","Type":"ContainerDied","Data":"db5100412d1df9310f04e3997778a5c3240caadd4be1940e1b6a5401c9d443d7"} Oct 31 01:46:17.912854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.912371 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db5100412d1df9310f04e3997778a5c3240caadd4be1940e1b6a5401c9d443d7" Oct 31 01:46:17.912854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.912430 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d" Oct 31 01:46:17.916085 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.916052 5756 generic.go:356] "Generic (PLEG): container finished" podID="fd6993e5-4f4b-42e8-b119-19c1c88bc46e" containerID="2380fb24bc4cba58915ebc08c321574dd6f15eccc0b305c021e8e9402088c771" exitCode=0 Oct 31 01:46:17.917186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:17.917109 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" event={"ID":"fd6993e5-4f4b-42e8-b119-19c1c88bc46e","Type":"ContainerDied","Data":"2380fb24bc4cba58915ebc08c321574dd6f15eccc0b305c021e8e9402088c771"} Oct 31 01:46:18.289888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.288694 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n"] Oct 31 01:46:18.327897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.325608 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fdf84559-aed8-4bbf-8770-296712aefb9e" containerName="revoke-keys" Oct 31 01:46:18.327897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.325640 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf84559-aed8-4bbf-8770-296712aefb9e" containerName="revoke-keys" Oct 31 01:46:18.327897 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.325800 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf84559-aed8-4bbf-8770-296712aefb9e" containerName="revoke-keys" Oct 31 01:46:18.358889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.356587 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n"] Oct 31 01:46:18.362887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.362166 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.545196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.544744 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.545196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.544826 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.545196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.544929 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjmqq\" (UniqueName: \"kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.646554 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.646515 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.646781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.646594 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.646781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.646663 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjmqq\" (UniqueName: \"kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.648117 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.648053 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.664990 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.650210 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.704704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.704664 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjmqq\" (UniqueName: \"kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq\") pod \"maas-api-revoke-sub-ade59137-4zv2n\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:18.827206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:18.827110 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:19.146045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.146011 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:19.182780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.182753 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca\") pod \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " Oct 31 01:46:19.182972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.182850 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca\") pod \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " Oct 31 01:46:19.182972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.182963 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l\" (UniqueName: \"kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l\") pod \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\" (UID: \"fd6993e5-4f4b-42e8-b119-19c1c88bc46e\") " Oct 31 01:46:19.183187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.183160 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca" pod "fd6993e5-4f4b-42e8-b119-19c1c88bc46e" (UID: "fd6993e5-4f4b-42e8-b119-19c1c88bc46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:19.183305 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.183296 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:19.185146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.185082 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca" pod "fd6993e5-4f4b-42e8-b119-19c1c88bc46e" (UID: "fd6993e5-4f4b-42e8-b119-19c1c88bc46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:19.185297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.185280 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l" pod "fd6993e5-4f4b-42e8-b119-19c1c88bc46e" (UID: "fd6993e5-4f4b-42e8-b119-19c1c88bc46e"). InnerVolumeSpecName "kube-api-access-h2h2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:19.280060 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:19.280031 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809e120e_134e_4ff4_9513_b0037c01066d.slice/crio-bf94dc0e575f9dd54380ad4cf958420a644a3e4ab0d662c96ad738fe2078c111 WatchSource:0}: Error finding container bf94dc0e575f9dd54380ad4cf958420a644a3e4ab0d662c96ad738fe2078c111: Status 404 returned error can't find the container with id bf94dc0e575f9dd54380ad4cf958420a644a3e4ab0d662c96ad738fe2078c111 Oct 31 01:46:19.284171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.284145 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2h2l\" (UniqueName: \"kubernetes.io/projected/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-kube-api-access-h2h2l\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:19.284268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.284174 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd6993e5-4f4b-42e8-b119-19c1c88bc46e-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:19.310268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.310241 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n"] Oct 31 01:46:19.930889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.930817 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" event={"ID":"809e120e-134e-4ff4-9513-b0037c01066d","Type":"ContainerStarted","Data":"2930526b9b1b6c63ed952a667e05edd6f6785ff9b1b833ba91e6ac271523ed1e"} Oct 31 01:46:19.931061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.930961 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" event={"ID":"809e120e-134e-4ff4-9513-b0037c01066d","Type":"ContainerStarted","Data":"bf94dc0e575f9dd54380ad4cf958420a644a3e4ab0d662c96ad738fe2078c111"} Oct 31 01:46:19.932418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.932399 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" Oct 31 01:46:19.932536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.932415 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp" event={"ID":"fd6993e5-4f4b-42e8-b119-19c1c88bc46e","Type":"ContainerDied","Data":"e5a95f74b9d404dbd40ad848d1cafbe03161aa1870963cded3620d627c542f0d"} Oct 31 01:46:19.932536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:19.932450 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a95f74b9d404dbd40ad848d1cafbe03161aa1870963cded3620d627c542f0d" Oct 31 01:46:20.095625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:20.095572 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" podStartSLOduration=2.095558079 podStartE2EDuration="2.095558079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:20.093358034 +0000 UTC m=+2951.900325577" watchObservedRunningTime="2026-10-31 01:46:20.095558079 +0000 UTC m=+2951.902525665" Oct 31 01:46:20.940400 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:20.940365 5756 generic.go:356] "Generic (PLEG): container finished" podID="809e120e-134e-4ff4-9513-b0037c01066d" containerID="2930526b9b1b6c63ed952a667e05edd6f6785ff9b1b833ba91e6ac271523ed1e" exitCode=0 Oct 31 01:46:20.940780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:20.940456 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" event={"ID":"809e120e-134e-4ff4-9513-b0037c01066d","Type":"ContainerDied","Data":"2930526b9b1b6c63ed952a667e05edd6f6785ff9b1b833ba91e6ac271523ed1e"} Oct 31 01:46:22.089482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.089450 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:22.115600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.115565 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca\") pod \"809e120e-134e-4ff4-9513-b0037c01066d\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " Oct 31 01:46:22.115769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.115650 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq\" (UniqueName: \"kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq\") pod \"809e120e-134e-4ff4-9513-b0037c01066d\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " Oct 31 01:46:22.115919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.115900 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca\") pod \"809e120e-134e-4ff4-9513-b0037c01066d\" (UID: \"809e120e-134e-4ff4-9513-b0037c01066d\") " Oct 31 01:46:22.116272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.116240 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca" pod "809e120e-134e-4ff4-9513-b0037c01066d" (UID: "809e120e-134e-4ff4-9513-b0037c01066d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:22.116529 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.116326 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:22.117951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.117925 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca" pod "809e120e-134e-4ff4-9513-b0037c01066d" (UID: "809e120e-134e-4ff4-9513-b0037c01066d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:22.118041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.117957 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq" pod "809e120e-134e-4ff4-9513-b0037c01066d" (UID: "809e120e-134e-4ff4-9513-b0037c01066d"). InnerVolumeSpecName "kube-api-access-tjmqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:22.217457 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.217382 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809e120e-134e-4ff4-9513-b0037c01066d-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:22.217457 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.217411 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjmqq\" (UniqueName: \"kubernetes.io/projected/809e120e-134e-4ff4-9513-b0037c01066d-kube-api-access-tjmqq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:22.557847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.557810 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28"] Oct 31 01:46:22.563630 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.563606 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-0c0723-trw28"] Oct 31 01:46:22.954954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.954873 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" Oct 31 01:46:22.954954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.954913 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n" event={"ID":"809e120e-134e-4ff4-9513-b0037c01066d","Type":"ContainerDied","Data":"bf94dc0e575f9dd54380ad4cf958420a644a3e4ab0d662c96ad738fe2078c111"} Oct 31 01:46:22.954954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:22.954955 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf94dc0e575f9dd54380ad4cf958420a644a3e4ab0d662c96ad738fe2078c111" Oct 31 01:46:24.317786 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:24.317750 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59111aa7-c90b-4a00-9891-49e28a26397a" path="/var/lib/kubelet/pods/59111aa7-c90b-4a00-9891-49e28a26397a/volumes" Oct 31 01:46:30.051445 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.051408 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp"] Oct 31 01:46:30.051912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.051898 5756 cpu_manager.go:414] "containerMap: removing container" podUID="809e120e-134e-4ff4-9513-b0037c01066d" containerName="revoke-keys" Oct 31 01:46:30.051965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.051913 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809e120e-134e-4ff4-9513-b0037c01066d" containerName="revoke-keys" Oct 31 01:46:30.051965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.051934 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fd6993e5-4f4b-42e8-b119-19c1c88bc46e" containerName="revoke-keys" Oct 31 01:46:30.051965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.051940 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd6993e5-4f4b-42e8-b119-19c1c88bc46e" containerName="revoke-keys" Oct 31 01:46:30.052069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.052034 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd6993e5-4f4b-42e8-b119-19c1c88bc46e" containerName="revoke-keys" Oct 31 01:46:30.052069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.052051 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="809e120e-134e-4ff4-9513-b0037c01066d" containerName="revoke-keys" Oct 31 01:46:30.055585 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.055571 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.110560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.110532 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njv96\" (UniqueName: \"kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.110740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.110623 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.110740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.110644 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.156714 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.156682 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:46:30.156920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.156759 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:46:30.159663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.159636 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp"] Oct 31 01:46:30.211481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.211447 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njv96\" (UniqueName: \"kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.211645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.211529 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.211645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.211549 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.212170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.212148 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.214110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.214088 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.268491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.268464 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njv96\" (UniqueName: \"kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96\") pod \"maas-api-revoke-sub-794c9efd-tj5tp\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.369036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.368962 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:30.744910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.744881 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp"] Oct 31 01:46:30.745767 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:30.745729 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918af203_6923_4d19_8514_4e59bd945ae5.slice/crio-dbeda5c85e2178fb257caff25eed9de1dae114ae3d1e393579f9b51e5ad24609 WatchSource:0}: Error finding container dbeda5c85e2178fb257caff25eed9de1dae114ae3d1e393579f9b51e5ad24609: Status 404 returned error can't find the container with id dbeda5c85e2178fb257caff25eed9de1dae114ae3d1e393579f9b51e5ad24609 Oct 31 01:46:30.914041 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.914002 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d"] Oct 31 01:46:30.914706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:30.914690 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-001a26-xk29d"] Oct 31 01:46:31.009889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:31.009777 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" event={"ID":"918af203-6923-4d19-8514-4e59bd945ae5","Type":"ContainerStarted","Data":"dbeda5c85e2178fb257caff25eed9de1dae114ae3d1e393579f9b51e5ad24609"} Oct 31 01:46:32.016902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:32.016784 5756 generic.go:356] "Generic (PLEG): container finished" podID="918af203-6923-4d19-8514-4e59bd945ae5" containerID="5d4ef9069ef22a12720be6d30d0f94a121ef840326f9de2f5e29c0701465bceb" exitCode=0 Oct 31 01:46:32.016902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:32.016893 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" event={"ID":"918af203-6923-4d19-8514-4e59bd945ae5","Type":"ContainerDied","Data":"5d4ef9069ef22a12720be6d30d0f94a121ef840326f9de2f5e29c0701465bceb"} Oct 31 01:46:32.066397 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:46:32.066360 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:46:32.311439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:32.311408 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf84559-aed8-4bbf-8770-296712aefb9e" path="/var/lib/kubelet/pods/fdf84559-aed8-4bbf-8770-296712aefb9e/volumes" Oct 31 01:46:33.173773 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.173751 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:33.244149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.244117 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca\") pod \"918af203-6923-4d19-8514-4e59bd945ae5\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " Oct 31 01:46:33.244351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.244335 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96\" (UniqueName: \"kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96\") pod \"918af203-6923-4d19-8514-4e59bd945ae5\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " Oct 31 01:46:33.244439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.244416 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca\") pod \"918af203-6923-4d19-8514-4e59bd945ae5\" (UID: \"918af203-6923-4d19-8514-4e59bd945ae5\") " Oct 31 01:46:33.244738 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.244717 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca" pod "918af203-6923-4d19-8514-4e59bd945ae5" (UID: "918af203-6923-4d19-8514-4e59bd945ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:33.246317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.246298 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca" pod "918af203-6923-4d19-8514-4e59bd945ae5" (UID: "918af203-6923-4d19-8514-4e59bd945ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:33.246520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.246494 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96" pod "918af203-6923-4d19-8514-4e59bd945ae5" (UID: "918af203-6923-4d19-8514-4e59bd945ae5"). InnerVolumeSpecName "kube-api-access-njv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:33.345787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.345741 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:33.345787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.345791 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/918af203-6923-4d19-8514-4e59bd945ae5-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:33.346009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:33.345807 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njv96\" (UniqueName: \"kubernetes.io/projected/918af203-6923-4d19-8514-4e59bd945ae5-kube-api-access-njv96\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:34.031915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:34.031886 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" Oct 31 01:46:34.031915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:34.031896 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp" event={"ID":"918af203-6923-4d19-8514-4e59bd945ae5","Type":"ContainerDied","Data":"dbeda5c85e2178fb257caff25eed9de1dae114ae3d1e393579f9b51e5ad24609"} Oct 31 01:46:34.032241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:34.031945 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbeda5c85e2178fb257caff25eed9de1dae114ae3d1e393579f9b51e5ad24609" Oct 31 01:46:35.109700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.109665 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp"] Oct 31 01:46:35.110198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.110183 5756 cpu_manager.go:414] "containerMap: removing container" podUID="918af203-6923-4d19-8514-4e59bd945ae5" containerName="revoke-keys" Oct 31 01:46:35.110246 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.110200 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="918af203-6923-4d19-8514-4e59bd945ae5" containerName="revoke-keys" Oct 31 01:46:35.110320 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.110311 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="918af203-6923-4d19-8514-4e59bd945ae5" containerName="revoke-keys" Oct 31 01:46:35.113683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.113665 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.215659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.215627 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:46:35.215659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.215629 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:46:35.220503 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.220477 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp"] Oct 31 01:46:35.269349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.269318 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrhp\" (UniqueName: \"kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.269502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.269385 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.269551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.269508 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.370556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.370471 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.370556 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.370548 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrhp\" (UniqueName: \"kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.370749 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.370599 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.371278 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.371259 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.373104 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.373085 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.426987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.426956 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrhp\" (UniqueName: \"kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp\") pod \"maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:35.726851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:35.726773 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:36.106578 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:36.106546 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp"] Oct 31 01:46:36.108074 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:36.108043 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6997dde6_3dbf_42f4_8b9b_26a5ce193b2a.slice/crio-1128603c53e5dbd6442e6dcc4ee031fb0bae335186b49740630a1c5668d25370 WatchSource:0}: Error finding container 1128603c53e5dbd6442e6dcc4ee031fb0bae335186b49740630a1c5668d25370: Status 404 returned error can't find the container with id 1128603c53e5dbd6442e6dcc4ee031fb0bae335186b49740630a1c5668d25370 Oct 31 01:46:37.051976 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:37.051946 5756 generic.go:356] "Generic (PLEG): container finished" podID="6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" containerID="200f99a0a4d77c6d1dff59e748148f973f51b4dc123a06187f3b9099a6ad91b3" exitCode=0 Oct 31 01:46:37.052349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:37.052026 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" event={"ID":"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a","Type":"ContainerDied","Data":"200f99a0a4d77c6d1dff59e748148f973f51b4dc123a06187f3b9099a6ad91b3"} Oct 31 01:46:37.052349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:37.052080 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" event={"ID":"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a","Type":"ContainerStarted","Data":"1128603c53e5dbd6442e6dcc4ee031fb0bae335186b49740630a1c5668d25370"} Oct 31 01:46:38.210128 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.210105 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:38.304133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.304103 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca\") pod \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " Oct 31 01:46:38.304341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.304205 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca\") pod \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " Oct 31 01:46:38.304419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.304355 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp\" (UniqueName: \"kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp\") pod \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\" (UID: \"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a\") " Oct 31 01:46:38.304535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.304512 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca" pod "6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" (UID: "6997dde6-3dbf-42f4-8b9b-26a5ce193b2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:38.304728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.304710 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:38.306423 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.306400 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca" pod "6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" (UID: "6997dde6-3dbf-42f4-8b9b-26a5ce193b2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:38.306536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.306451 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp" pod "6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" (UID: "6997dde6-3dbf-42f4-8b9b-26a5ce193b2a"). InnerVolumeSpecName "kube-api-access-ddrhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:38.405527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.405492 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:38.405527 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:38.405530 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrhp\" (UniqueName: \"kubernetes.io/projected/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a-kube-api-access-ddrhp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:39.066550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:39.066518 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" Oct 31 01:46:39.066741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:39.066519 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp" event={"ID":"6997dde6-3dbf-42f4-8b9b-26a5ce193b2a","Type":"ContainerDied","Data":"1128603c53e5dbd6442e6dcc4ee031fb0bae335186b49740630a1c5668d25370"} Oct 31 01:46:39.066741 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:39.066680 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1128603c53e5dbd6442e6dcc4ee031fb0bae335186b49740630a1c5668d25370" Oct 31 01:46:40.785670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.785635 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp"] Oct 31 01:46:40.786232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.786215 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" containerName="revoke-keys" Oct 31 01:46:40.786277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.786233 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" containerName="revoke-keys" Oct 31 01:46:40.786335 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.786325 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" containerName="revoke-keys" Oct 31 01:46:40.789831 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.789813 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:40.841731 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.841707 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:46:40.891398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.891361 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:46:40.894254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.894229 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp"] Oct 31 01:46:40.934481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.934448 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:40.934632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.934497 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vw8\" (UniqueName: \"kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:40.934699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:40.934658 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.036210 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.036128 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.036377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.036299 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.036377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.036348 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vw8\" (UniqueName: \"kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.036789 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.036768 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.038850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.038827 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.092452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.092424 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vw8\" (UniqueName: \"kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8\") pod \"maas-api-revoke-sub-3936f42e-nf8qp\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.103317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.103290 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:41.479913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:41.479884 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp"] Oct 31 01:46:41.481505 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:41.481473 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039d23d7_fe0b_4ab0_9cd6_845b76a3853c.slice/crio-9796c25ee2104857f5ccd1b0ac3580effc868f81286921b630b6dceeddc9543d WatchSource:0}: Error finding container 9796c25ee2104857f5ccd1b0ac3580effc868f81286921b630b6dceeddc9543d: Status 404 returned error can't find the container with id 9796c25ee2104857f5ccd1b0ac3580effc868f81286921b630b6dceeddc9543d Oct 31 01:46:42.088028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:42.087989 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" event={"ID":"039d23d7-fe0b-4ab0-9cd6-845b76a3853c","Type":"ContainerStarted","Data":"27c3f497fa0a202f7ef4859d75e2ed4218152044e12a8cc6537523bf0b0688b0"} Oct 31 01:46:42.088479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:42.088047 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" event={"ID":"039d23d7-fe0b-4ab0-9cd6-845b76a3853c","Type":"ContainerStarted","Data":"9796c25ee2104857f5ccd1b0ac3580effc868f81286921b630b6dceeddc9543d"} Oct 31 01:46:42.204546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:42.204489 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" podStartSLOduration=2.20447448 podStartE2EDuration="2.20447448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:42.201761252 +0000 UTC m=+2974.008728786" watchObservedRunningTime="2026-10-31 01:46:42.20447448 +0000 UTC m=+2974.011442022" Oct 31 01:46:43.123968 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:43.123930 5756 generic.go:356] "Generic (PLEG): container finished" podID="039d23d7-fe0b-4ab0-9cd6-845b76a3853c" containerID="27c3f497fa0a202f7ef4859d75e2ed4218152044e12a8cc6537523bf0b0688b0" exitCode=0 Oct 31 01:46:43.124832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:43.124606 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" event={"ID":"039d23d7-fe0b-4ab0-9cd6-845b76a3853c","Type":"ContainerDied","Data":"27c3f497fa0a202f7ef4859d75e2ed4218152044e12a8cc6537523bf0b0688b0"} Oct 31 01:46:44.429406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.429380 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:44.493014 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.492919 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca\") pod \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " Oct 31 01:46:44.493014 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.492978 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca\") pod \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " Oct 31 01:46:44.493276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.493088 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8\" (UniqueName: \"kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8\") pod \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\" (UID: \"039d23d7-fe0b-4ab0-9cd6-845b76a3853c\") " Oct 31 01:46:44.493396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.493372 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca" pod "039d23d7-fe0b-4ab0-9cd6-845b76a3853c" (UID: "039d23d7-fe0b-4ab0-9cd6-845b76a3853c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:44.493577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.493558 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:44.495440 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.495408 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8" pod "039d23d7-fe0b-4ab0-9cd6-845b76a3853c" (UID: "039d23d7-fe0b-4ab0-9cd6-845b76a3853c"). InnerVolumeSpecName "kube-api-access-w7vw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:44.495562 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.495482 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca" pod "039d23d7-fe0b-4ab0-9cd6-845b76a3853c" (UID: "039d23d7-fe0b-4ab0-9cd6-845b76a3853c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:44.594603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.594570 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:44.594603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:44.594597 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vw8\" (UniqueName: \"kubernetes.io/projected/039d23d7-fe0b-4ab0-9cd6-845b76a3853c-kube-api-access-w7vw8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:45.167929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.167890 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" event={"ID":"039d23d7-fe0b-4ab0-9cd6-845b76a3853c","Type":"ContainerDied","Data":"9796c25ee2104857f5ccd1b0ac3580effc868f81286921b630b6dceeddc9543d"} Oct 31 01:46:45.167929 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.167912 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp" Oct 31 01:46:45.168169 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.167940 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9796c25ee2104857f5ccd1b0ac3580effc868f81286921b630b6dceeddc9543d" Oct 31 01:46:45.449344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.449212 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q"] Oct 31 01:46:45.449837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.449773 5756 cpu_manager.go:414] "containerMap: removing container" podUID="039d23d7-fe0b-4ab0-9cd6-845b76a3853c" containerName="revoke-keys" Oct 31 01:46:45.449837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.449787 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="039d23d7-fe0b-4ab0-9cd6-845b76a3853c" containerName="revoke-keys" Oct 31 01:46:45.449987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.449907 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="039d23d7-fe0b-4ab0-9cd6-845b76a3853c" containerName="revoke-keys" Oct 31 01:46:45.453248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.453229 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.504045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.503992 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.504287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.504174 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkfgz\" (UniqueName: \"kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.504369 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.504286 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.504913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.504837 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:46:45.505081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.504920 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:46:45.507547 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.507524 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q"] Oct 31 01:46:45.605330 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.605296 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.605510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.605361 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkfgz\" (UniqueName: \"kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.605510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.605397 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.605948 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.605925 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.607817 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.607793 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.661676 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.661648 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkfgz\" (UniqueName: \"kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz\") pod \"maas-api-revoke-sub-0ac870d2-k7g5q\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:45.771083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:45.770998 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:46.153226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:46.153194 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q"] Oct 31 01:46:46.154000 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:46.153975 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3177ea16_91bb_41ba_85ca_f51ce70dd71a.slice/crio-b1880092880821330a402a936acebada8a0d1c2f0d58260c5a433b8f10dd0d17 WatchSource:0}: Error finding container b1880092880821330a402a936acebada8a0d1c2f0d58260c5a433b8f10dd0d17: Status 404 returned error can't find the container with id b1880092880821330a402a936acebada8a0d1c2f0d58260c5a433b8f10dd0d17 Oct 31 01:46:46.177305 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:46.177227 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" event={"ID":"3177ea16-91bb-41ba-85ca-f51ce70dd71a","Type":"ContainerStarted","Data":"b1880092880821330a402a936acebada8a0d1c2f0d58260c5a433b8f10dd0d17"} Oct 31 01:46:47.187071 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:47.187036 5756 generic.go:356] "Generic (PLEG): container finished" podID="3177ea16-91bb-41ba-85ca-f51ce70dd71a" containerID="44d08612a3740b5487ddb7294cbae2e5f8350670d7b6acb4206e64e1baf8faca" exitCode=0 Oct 31 01:46:47.187462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:47.187116 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" event={"ID":"3177ea16-91bb-41ba-85ca-f51ce70dd71a","Type":"ContainerDied","Data":"44d08612a3740b5487ddb7294cbae2e5f8350670d7b6acb4206e64e1baf8faca"} Oct 31 01:46:48.352875 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.352835 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:48.436267 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.436225 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca\") pod \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " Oct 31 01:46:48.436524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.436311 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz\" (UniqueName: \"kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz\") pod \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " Oct 31 01:46:48.436524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.436366 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca\") pod \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\" (UID: \"3177ea16-91bb-41ba-85ca-f51ce70dd71a\") " Oct 31 01:46:48.437245 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.437209 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca" pod "3177ea16-91bb-41ba-85ca-f51ce70dd71a" (UID: "3177ea16-91bb-41ba-85ca-f51ce70dd71a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:48.440450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.440419 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:48.441687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.441652 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz" pod "3177ea16-91bb-41ba-85ca-f51ce70dd71a" (UID: "3177ea16-91bb-41ba-85ca-f51ce70dd71a"). InnerVolumeSpecName "kube-api-access-dkfgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:48.442088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.442051 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca" pod "3177ea16-91bb-41ba-85ca-f51ce70dd71a" (UID: "3177ea16-91bb-41ba-85ca-f51ce70dd71a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:48.492048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.492002 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl"] Oct 31 01:46:48.492546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.492528 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3177ea16-91bb-41ba-85ca-f51ce70dd71a" containerName="revoke-keys" Oct 31 01:46:48.492655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.492550 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3177ea16-91bb-41ba-85ca-f51ce70dd71a" containerName="revoke-keys" Oct 31 01:46:48.492763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.492750 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3177ea16-91bb-41ba-85ca-f51ce70dd71a" containerName="revoke-keys" Oct 31 01:46:48.496315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.496296 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.541366 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.541338 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.541528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.541466 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xk4c\" (UniqueName: \"kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.541528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.541513 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.541627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.541558 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3177ea16-91bb-41ba-85ca-f51ce70dd71a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:48.541627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.541575 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkfgz\" (UniqueName: \"kubernetes.io/projected/3177ea16-91bb-41ba-85ca-f51ce70dd71a-kube-api-access-dkfgz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:48.550681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.550654 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl"] Oct 31 01:46:48.642845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.642813 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.643023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.642911 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.643023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.642994 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xk4c\" (UniqueName: \"kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.643443 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.643422 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.645386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.645361 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.699297 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.699267 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xk4c\" (UniqueName: \"kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c\") pod \"maas-api-revoke-sub-8f12c611-2vbnl\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:48.810501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:48.810470 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:49.184177 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:49.184150 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl"] Oct 31 01:46:49.185670 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:49.185635 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1984ee3a_c2a5_4f5c_861c_d30ac881cf86.slice/crio-5d349c2c53e00e1e977b981cdaf17bcb7a6fa6c98cce5b18cff8bd9d4aaab24b WatchSource:0}: Error finding container 5d349c2c53e00e1e977b981cdaf17bcb7a6fa6c98cce5b18cff8bd9d4aaab24b: Status 404 returned error can't find the container with id 5d349c2c53e00e1e977b981cdaf17bcb7a6fa6c98cce5b18cff8bd9d4aaab24b Oct 31 01:46:49.204057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:49.203789 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" Oct 31 01:46:49.204057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:49.203818 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q" event={"ID":"3177ea16-91bb-41ba-85ca-f51ce70dd71a","Type":"ContainerDied","Data":"b1880092880821330a402a936acebada8a0d1c2f0d58260c5a433b8f10dd0d17"} Oct 31 01:46:49.204057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:49.203883 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1880092880821330a402a936acebada8a0d1c2f0d58260c5a433b8f10dd0d17" Oct 31 01:46:49.205260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:49.205233 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" event={"ID":"1984ee3a-c2a5-4f5c-861c-d30ac881cf86","Type":"ContainerStarted","Data":"5d349c2c53e00e1e977b981cdaf17bcb7a6fa6c98cce5b18cff8bd9d4aaab24b"} Oct 31 01:46:50.213489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:50.213455 5756 generic.go:356] "Generic (PLEG): container finished" podID="1984ee3a-c2a5-4f5c-861c-d30ac881cf86" containerID="b2c81eb27a2e03e583a8dc5f8f286d3711a2737e2cf2ecfc2054c8ddce39a582" exitCode=0 Oct 31 01:46:50.213930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:50.213520 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" event={"ID":"1984ee3a-c2a5-4f5c-861c-d30ac881cf86","Type":"ContainerDied","Data":"b2c81eb27a2e03e583a8dc5f8f286d3711a2737e2cf2ecfc2054c8ddce39a582"} Oct 31 01:46:51.368350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.368323 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:51.473569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.473488 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c\" (UniqueName: \"kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c\") pod \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " Oct 31 01:46:51.473569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.473548 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca\") pod \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " Oct 31 01:46:51.473796 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.473693 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca\") pod \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\" (UID: \"1984ee3a-c2a5-4f5c-861c-d30ac881cf86\") " Oct 31 01:46:51.474100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.474072 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca" pod "1984ee3a-c2a5-4f5c-861c-d30ac881cf86" (UID: "1984ee3a-c2a5-4f5c-861c-d30ac881cf86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:51.475937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.475911 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c" pod "1984ee3a-c2a5-4f5c-861c-d30ac881cf86" (UID: "1984ee3a-c2a5-4f5c-861c-d30ac881cf86"). InnerVolumeSpecName "kube-api-access-8xk4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:51.475937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.475916 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca" pod "1984ee3a-c2a5-4f5c-861c-d30ac881cf86" (UID: "1984ee3a-c2a5-4f5c-861c-d30ac881cf86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:51.575349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.575315 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:51.575349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.575343 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xk4c\" (UniqueName: \"kubernetes.io/projected/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-kube-api-access-8xk4c\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:51.575349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:51.575354 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1984ee3a-c2a5-4f5c-861c-d30ac881cf86-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:46:52.228857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:52.228826 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" Oct 31 01:46:52.229045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:52.228885 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl" event={"ID":"1984ee3a-c2a5-4f5c-861c-d30ac881cf86","Type":"ContainerDied","Data":"5d349c2c53e00e1e977b981cdaf17bcb7a6fa6c98cce5b18cff8bd9d4aaab24b"} Oct 31 01:46:52.229045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:52.228973 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d349c2c53e00e1e977b981cdaf17bcb7a6fa6c98cce5b18cff8bd9d4aaab24b" Oct 31 01:46:58.119268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.119234 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n"] Oct 31 01:46:58.119941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.119923 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1984ee3a-c2a5-4f5c-861c-d30ac881cf86" containerName="revoke-keys" Oct 31 01:46:58.119994 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.119942 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1984ee3a-c2a5-4f5c-861c-d30ac881cf86" containerName="revoke-keys" Oct 31 01:46:58.120049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.120027 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1984ee3a-c2a5-4f5c-861c-d30ac881cf86" containerName="revoke-keys" Oct 31 01:46:58.123723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.123695 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.183482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.183452 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:46:58.233508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.233479 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:46:58.240322 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.240287 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n"] Oct 31 01:46:58.256291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.255797 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.256291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.255901 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.256291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.255933 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthnw\" (UniqueName: \"kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.356790 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.356754 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.356994 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.356816 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.356994 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.356838 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthnw\" (UniqueName: \"kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.357604 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.357575 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.359242 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.359222 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.413972 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.413893 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthnw\" (UniqueName: \"kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw\") pod \"maas-api-revoke-sub-9dae6b7d-gk65n\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.437675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.437643 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:46:58.818437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:58.818405 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n"] Oct 31 01:46:58.819819 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:46:58.819180 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715d5105_69c6_4587_be86_3404d1ca113c.slice/crio-d29d20746590cd85943e1445e488937098bde487bb996a14a9247b0fcd72130e WatchSource:0}: Error finding container d29d20746590cd85943e1445e488937098bde487bb996a14a9247b0fcd72130e: Status 404 returned error can't find the container with id d29d20746590cd85943e1445e488937098bde487bb996a14a9247b0fcd72130e Oct 31 01:46:59.309886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:59.308231 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" event={"ID":"715d5105-69c6-4587-be86-3404d1ca113c","Type":"ContainerStarted","Data":"44facac5c627f0f6c733dcc1d9d9ec09e27f020dbd5bb3275ed2cd7a3dcb4de2"} Oct 31 01:46:59.309886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:59.308290 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" event={"ID":"715d5105-69c6-4587-be86-3404d1ca113c","Type":"ContainerStarted","Data":"d29d20746590cd85943e1445e488937098bde487bb996a14a9247b0fcd72130e"} Oct 31 01:46:59.423837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:46:59.423737 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" podStartSLOduration=1.423724762 podStartE2EDuration="1.423724762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:59.420692879 +0000 UTC m=+2991.227660423" watchObservedRunningTime="2026-10-31 01:46:59.423724762 +0000 UTC m=+2991.230692358" Oct 31 01:47:00.316729 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:00.316696 5756 generic.go:356] "Generic (PLEG): container finished" podID="715d5105-69c6-4587-be86-3404d1ca113c" containerID="44facac5c627f0f6c733dcc1d9d9ec09e27f020dbd5bb3275ed2cd7a3dcb4de2" exitCode=0 Oct 31 01:47:00.317157 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:00.316780 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" event={"ID":"715d5105-69c6-4587-be86-3404d1ca113c","Type":"ContainerDied","Data":"44facac5c627f0f6c733dcc1d9d9ec09e27f020dbd5bb3275ed2cd7a3dcb4de2"} Oct 31 01:47:01.530904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.530604 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:47:01.595997 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.595903 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca\") pod \"715d5105-69c6-4587-be86-3404d1ca113c\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " Oct 31 01:47:01.596179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.596060 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca\") pod \"715d5105-69c6-4587-be86-3404d1ca113c\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " Oct 31 01:47:01.596179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.596138 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw\" (UniqueName: \"kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw\") pod \"715d5105-69c6-4587-be86-3404d1ca113c\" (UID: \"715d5105-69c6-4587-be86-3404d1ca113c\") " Oct 31 01:47:01.596378 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.596342 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca" pod "715d5105-69c6-4587-be86-3404d1ca113c" (UID: "715d5105-69c6-4587-be86-3404d1ca113c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:01.596620 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.596595 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:47:01.598785 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.598757 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw" pod "715d5105-69c6-4587-be86-3404d1ca113c" (UID: "715d5105-69c6-4587-be86-3404d1ca113c"). InnerVolumeSpecName "kube-api-access-jthnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:01.600877 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.600828 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca" pod "715d5105-69c6-4587-be86-3404d1ca113c" (UID: "715d5105-69c6-4587-be86-3404d1ca113c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:01.698424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.698390 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/715d5105-69c6-4587-be86-3404d1ca113c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:47:01.698424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:01.698426 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthnw\" (UniqueName: \"kubernetes.io/projected/715d5105-69c6-4587-be86-3404d1ca113c-kube-api-access-jthnw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:47:02.070742 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:47:02.070700 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:47:02.331834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:02.331753 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" Oct 31 01:47:02.331834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:02.331794 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n" event={"ID":"715d5105-69c6-4587-be86-3404d1ca113c","Type":"ContainerDied","Data":"d29d20746590cd85943e1445e488937098bde487bb996a14a9247b0fcd72130e"} Oct 31 01:47:02.332096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:02.331842 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d29d20746590cd85943e1445e488937098bde487bb996a14a9247b0fcd72130e" Oct 31 01:47:06.189142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.189104 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f"] Oct 31 01:47:06.189634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.189621 5756 cpu_manager.go:414] "containerMap: removing container" podUID="715d5105-69c6-4587-be86-3404d1ca113c" containerName="revoke-keys" Oct 31 01:47:06.189680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.189636 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715d5105-69c6-4587-be86-3404d1ca113c" containerName="revoke-keys" Oct 31 01:47:06.189723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.189714 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="715d5105-69c6-4587-be86-3404d1ca113c" containerName="revoke-keys" Oct 31 01:47:06.193021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.193006 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.248225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.248195 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f"] Oct 31 01:47:06.294291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.294259 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-41014a\"" Oct 31 01:47:06.294617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.294362 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-41014a-dockercfg-zhl9c\"" Oct 31 01:47:06.352733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.352695 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r298x\" (UniqueName: \"kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x\") pod \"payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.352936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.352836 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.355023 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.354998 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd"] Oct 31 01:47:06.359374 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.359354 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.413980 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.413938 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd"] Oct 31 01:47:06.453670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.453589 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r298x\" (UniqueName: \"kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x\") pod \"payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.453851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.453766 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.453989 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.453855 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psqs8\" (UniqueName: \"kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8\") pod \"payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.454057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.453995 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.454569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.454548 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.515725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.515689 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r298x\" (UniqueName: \"kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x\") pod \"payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:06.555085 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.555055 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psqs8\" (UniqueName: \"kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8\") pod \"payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.555256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.555115 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.556003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.555976 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.611826 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.611791 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psqs8\" (UniqueName: \"kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8\") pod \"payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.672172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.672129 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:06.806208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:06.806181 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:07.055309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.055279 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd"] Oct 31 01:47:07.056061 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:47:07.056032 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73fc1fe_ef05_4cff_ad53_52ebf7588186.slice/crio-2796aec83f7914ad347912b40700be6c1724281570f124561dbabdbaa8232560 WatchSource:0}: Error finding container 2796aec83f7914ad347912b40700be6c1724281570f124561dbabdbaa8232560: Status 404 returned error can't find the container with id 2796aec83f7914ad347912b40700be6c1724281570f124561dbabdbaa8232560 Oct 31 01:47:07.186639 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.186610 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f"] Oct 31 01:47:07.187493 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:47:07.187454 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafdcf97a_56f4_4b9e_a347_1296bc20c7bf.slice/crio-95bdd0d84aa47e88e995d3cd99f605c038d55b9923226f4d41526a772c731a6d WatchSource:0}: Error finding container 95bdd0d84aa47e88e995d3cd99f605c038d55b9923226f4d41526a772c731a6d: Status 404 returned error can't find the container with id 95bdd0d84aa47e88e995d3cd99f605c038d55b9923226f4d41526a772c731a6d Oct 31 01:47:07.369275 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.369186 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" event={"ID":"f73fc1fe-ef05-4cff-ad53-52ebf7588186","Type":"ContainerStarted","Data":"5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4"} Oct 31 01:47:07.369275 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.369252 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" event={"ID":"f73fc1fe-ef05-4cff-ad53-52ebf7588186","Type":"ContainerStarted","Data":"2796aec83f7914ad347912b40700be6c1724281570f124561dbabdbaa8232560"} Oct 31 01:47:07.369786 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.369284 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:07.370904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.370856 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" event={"ID":"afdcf97a-56f4-4b9e-a347-1296bc20c7bf","Type":"ContainerStarted","Data":"0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688"} Oct 31 01:47:07.371034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.370935 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:07.371034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.370964 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" event={"ID":"afdcf97a-56f4-4b9e-a347-1296bc20c7bf","Type":"ContainerStarted","Data":"95bdd0d84aa47e88e995d3cd99f605c038d55b9923226f4d41526a772c731a6d"} Oct 31 01:47:07.488355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.488297 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" podStartSLOduration=1.488279081 podStartE2EDuration="1.488279081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:07.487551065 +0000 UTC m=+2999.294518618" watchObservedRunningTime="2026-10-31 01:47:07.488279081 +0000 UTC m=+2999.295246622" Oct 31 01:47:07.605631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:07.605564 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" podStartSLOduration=1.605544687 podStartE2EDuration="1.605544687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:07.600894286 +0000 UTC m=+2999.407861836" watchObservedRunningTime="2026-10-31 01:47:07.605544687 +0000 UTC m=+2999.412512231" Oct 31 01:47:08.659287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.659256 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx"] Oct 31 01:47:08.663493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.663473 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.768594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.768554 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx"] Oct 31 01:47:08.780713 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.780653 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.780898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.780729 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.780898 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.780778 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.781046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.780905 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.781107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.781055 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.781107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.781097 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.781265 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.781231 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.781372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.781270 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmvs\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.781434 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.781387 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.882800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.882767 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.882800 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.882798 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.882827 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.882855 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.882982 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883052 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883016 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883078 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883101 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmvs\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883348 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883155 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883507 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883551 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883587 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883826 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.883936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.883889 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.885363 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.885344 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.885493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.885478 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.942267 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.942196 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmvs\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.943349 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.943315 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:08.977705 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:08.977680 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:09.407056 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:09.407029 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx"] Oct 31 01:47:09.413533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:09.413499 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:47:09.413670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:09.413578 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:47:09.413670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:09.413620 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:47:10.299222 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.299194 5756 scope.go:122] "RemoveContainer" containerID="57f4a9089a36b6bb3a8393151b595ea7fac870cdf6595f2d0b20c4674a6cd81e" Oct 31 01:47:10.396493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.396446 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" event={"ID":"092f149c-9821-422c-8d26-aafeb22ac5ea","Type":"ContainerStarted","Data":"29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635"} Oct 31 01:47:10.396700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.396553 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" event={"ID":"092f149c-9821-422c-8d26-aafeb22ac5ea","Type":"ContainerStarted","Data":"94e964407eb515606a77c9d364647ff3f4ca92f71b306a10c235ed9d9e0b7d26"} Oct 31 01:47:10.573185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.573079 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podStartSLOduration=2.573064622 podStartE2EDuration="2.573064622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:10.56132545 +0000 UTC m=+3002.368292994" watchObservedRunningTime="2026-10-31 01:47:10.573064622 +0000 UTC m=+3002.380032166" Oct 31 01:47:10.962986 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.962848 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:47:10.978254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.978227 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:10.979656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.979630 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:47:10.979787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:10.979678 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:47:11.978628 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:11.978594 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:47:11.979079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:11.978650 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:47:12.978352 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:12.978316 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:47:12.978642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:12.978613 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:47:13.978499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:13.978466 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:47:13.978679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:13.978519 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:47:14.979076 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:14.979040 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:47:14.979555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:14.979100 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:47:15.978293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:15.978258 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 31 01:47:15.978505 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:15.978315 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 31 01:47:17.950806 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:17.950773 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:17.951280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:17.951229 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:17.952638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:17.952611 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:47:18.380293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:18.380262 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 01:47:18.380475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:18.380399 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 01:47:28.751610 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:28.751577 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp"] Oct 31 01:47:28.801886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:28.801833 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-76aba6-hq2kp"] Oct 31 01:47:30.311847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:30.311803 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6997dde6-3dbf-42f4-8b9b-26a5ce193b2a" path="/var/lib/kubelet/pods/6997dde6-3dbf-42f4-8b9b-26a5ce193b2a/volumes" Oct 31 01:47:32.066259 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:47:32.066223 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:47:39.962461 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:39.962406 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d"] Oct 31 01:47:39.988886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:39.988531 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.050386 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.049712 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-dc9e63\"" Oct 31 01:47:40.107059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.107020 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d"] Oct 31 01:47:40.110784 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.110757 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj"] Oct 31 01:47:40.114953 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.114937 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.119388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.119359 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc958\" (UniqueName: \"kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958\") pod \"payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.119583 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.119559 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.180883 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.180279 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-dc9e63-dockercfg-z2psv\"" Oct 31 01:47:40.199954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.196475 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj"] Oct 31 01:47:40.221280 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.221191 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.221444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.221295 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.221444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.221379 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc958\" (UniqueName: \"kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958\") pod \"payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.221444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.221399 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z65vm\" (UniqueName: \"kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm\") pod \"payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.222136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.222114 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.295706 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.295670 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc958\" (UniqueName: \"kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958\") pod \"payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.314122 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.314093 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:40.322566 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.322534 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z65vm\" (UniqueName: \"kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm\") pod \"payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.322699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.322589 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.323329 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.323306 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.380188 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.380155 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z65vm\" (UniqueName: \"kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm\") pod \"payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.434663 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.434627 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:40.723164 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.723138 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d"] Oct 31 01:47:40.724639 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:47:40.724609 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9dec466_5822_41ca_ad30_4a8b3ba2b90b.slice/crio-4abcd13bc0e2084d18af1be91209c90126fbfa34cfe753c33ecdb92248840943 WatchSource:0}: Error finding container 4abcd13bc0e2084d18af1be91209c90126fbfa34cfe753c33ecdb92248840943: Status 404 returned error can't find the container with id 4abcd13bc0e2084d18af1be91209c90126fbfa34cfe753c33ecdb92248840943 Oct 31 01:47:40.832048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:40.832021 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj"] Oct 31 01:47:40.833988 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:47:40.833952 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc606c_1581_416c_ae52_67ead76bd713.slice/crio-4d635c6e888def66f5ca2e0dd3f2f70720662c959ad981e61eee5b47af72cbe9 WatchSource:0}: Error finding container 4d635c6e888def66f5ca2e0dd3f2f70720662c959ad981e61eee5b47af72cbe9: Status 404 returned error can't find the container with id 4d635c6e888def66f5ca2e0dd3f2f70720662c959ad981e61eee5b47af72cbe9 Oct 31 01:47:41.682232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.682195 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" event={"ID":"e9dec466-5822-41ca-ad30-4a8b3ba2b90b","Type":"ContainerStarted","Data":"563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206"} Oct 31 01:47:41.682669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.682254 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" event={"ID":"e9dec466-5822-41ca-ad30-4a8b3ba2b90b","Type":"ContainerStarted","Data":"4abcd13bc0e2084d18af1be91209c90126fbfa34cfe753c33ecdb92248840943"} Oct 31 01:47:41.682669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.682285 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:41.684061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.684034 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" event={"ID":"f1fc606c-1581-416c-ae52-67ead76bd713","Type":"ContainerStarted","Data":"91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7"} Oct 31 01:47:41.684180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.684082 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" event={"ID":"f1fc606c-1581-416c-ae52-67ead76bd713","Type":"ContainerStarted","Data":"4d635c6e888def66f5ca2e0dd3f2f70720662c959ad981e61eee5b47af72cbe9"} Oct 31 01:47:41.684180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.684120 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:41.789919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.789884 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c"] Oct 31 01:47:41.797004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.796953 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" podStartSLOduration=2.7969395 podStartE2EDuration="2.7969395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:41.79579616 +0000 UTC m=+3033.602763716" watchObservedRunningTime="2026-10-31 01:47:41.7969395 +0000 UTC m=+3033.603907054" Oct 31 01:47:41.797685 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.797666 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.828423 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.828379 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c"] Oct 31 01:47:41.840952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.840918 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksx4h\" (UniqueName: \"kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.841099 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.841067 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.841167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.841139 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.849105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.849080 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:47:41.849241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.849121 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:47:41.911471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.911420 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" podStartSLOduration=1.911408088 podStartE2EDuration="1.911408088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:41.90824048 +0000 UTC m=+3033.715208024" watchObservedRunningTime="2026-10-31 01:47:41.911408088 +0000 UTC m=+3033.718375630" Oct 31 01:47:41.942603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.942528 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksx4h\" (UniqueName: \"kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.942748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.942610 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.942748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.942667 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.943282 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.943256 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:41.945105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:41.945079 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:42.000616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:42.000584 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksx4h\" (UniqueName: \"kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h\") pod \"maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:42.111644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:42.111615 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:42.493593 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:42.493561 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c"] Oct 31 01:47:42.494478 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:47:42.494448 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce286ca_b87f_41dc_aa65_efa45a5f139a.slice/crio-5666a4036e9875bb298b395bffc659818053d5529160363b37c366e05dcdab77 WatchSource:0}: Error finding container 5666a4036e9875bb298b395bffc659818053d5529160363b37c366e05dcdab77: Status 404 returned error can't find the container with id 5666a4036e9875bb298b395bffc659818053d5529160363b37c366e05dcdab77 Oct 31 01:47:42.691934 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:42.691896 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" event={"ID":"2ce286ca-b87f-41dc-aa65-efa45a5f139a","Type":"ContainerStarted","Data":"5666a4036e9875bb298b395bffc659818053d5529160363b37c366e05dcdab77"} Oct 31 01:47:43.700586 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:43.700554 5756 generic.go:356] "Generic (PLEG): container finished" podID="2ce286ca-b87f-41dc-aa65-efa45a5f139a" containerID="981386f0d15119fa968cba9d6848bae78af94675205a8442887bfb34c148eaeb" exitCode=0 Oct 31 01:47:43.700976 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:43.700609 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" event={"ID":"2ce286ca-b87f-41dc-aa65-efa45a5f139a","Type":"ContainerDied","Data":"981386f0d15119fa968cba9d6848bae78af94675205a8442887bfb34c148eaeb"} Oct 31 01:47:44.859341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.859308 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:44.980473 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.980386 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca\") pod \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " Oct 31 01:47:44.980648 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.980516 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h\" (UniqueName: \"kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h\") pod \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " Oct 31 01:47:44.980719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.980697 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca\") pod \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\" (UID: \"2ce286ca-b87f-41dc-aa65-efa45a5f139a\") " Oct 31 01:47:44.981088 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.981063 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca" pod "2ce286ca-b87f-41dc-aa65-efa45a5f139a" (UID: "2ce286ca-b87f-41dc-aa65-efa45a5f139a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:44.982667 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.982637 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca" pod "2ce286ca-b87f-41dc-aa65-efa45a5f139a" (UID: "2ce286ca-b87f-41dc-aa65-efa45a5f139a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:44.982762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:44.982700 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h" pod "2ce286ca-b87f-41dc-aa65-efa45a5f139a" (UID: "2ce286ca-b87f-41dc-aa65-efa45a5f139a"). InnerVolumeSpecName "kube-api-access-ksx4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:45.082497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:45.082461 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:47:45.082497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:45.082493 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ce286ca-b87f-41dc-aa65-efa45a5f139a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:47:45.082497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:45.082506 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksx4h\" (UniqueName: \"kubernetes.io/projected/2ce286ca-b87f-41dc-aa65-efa45a5f139a-kube-api-access-ksx4h\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:47:45.715012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:45.714974 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" event={"ID":"2ce286ca-b87f-41dc-aa65-efa45a5f139a","Type":"ContainerDied","Data":"5666a4036e9875bb298b395bffc659818053d5529160363b37c366e05dcdab77"} Oct 31 01:47:45.715196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:45.715025 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5666a4036e9875bb298b395bffc659818053d5529160363b37c366e05dcdab77" Oct 31 01:47:45.715196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:45.714992 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c" Oct 31 01:47:52.694873 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:52.694829 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 01:47:52.695275 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:52.694943 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 01:47:54.202911 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.202855 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn"] Oct 31 01:47:54.203455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.203441 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2ce286ca-b87f-41dc-aa65-efa45a5f139a" containerName="revoke-keys" Oct 31 01:47:54.203501 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.203457 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce286ca-b87f-41dc-aa65-efa45a5f139a" containerName="revoke-keys" Oct 31 01:47:54.203560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.203550 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce286ca-b87f-41dc-aa65-efa45a5f139a" containerName="revoke-keys" Oct 31 01:47:54.207209 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.207188 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.273027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.272991 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn"] Oct 31 01:47:54.294522 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294479 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.294680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294598 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.294680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294670 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.294760 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294720 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.294820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294803 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.294853 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294845 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.294926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.294895 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.295038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.295019 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.295077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.295061 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9ck6\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396215 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396269 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396325 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396340 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396365 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396405 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396644 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396709 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396718 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396803 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396819 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.396904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.396840 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9ck6\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.397068 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.397021 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.397682 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.397660 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.398927 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.398907 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.399027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.398943 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.452873 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.452825 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.453078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.453033 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9ck6\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6\") pod \"e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.520787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.520750 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:54.949875 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.949831 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn"] Oct 31 01:47:54.987509 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:47:54.987483 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eeb0e8f_51a3_4608_a56a_f50e7ba325ad.slice/crio-0c45b1a07f0809c5b7dc33d8aa61bc9d1eb0e0f35c79c558a446e2bbfca6e422 WatchSource:0}: Error finding container 0c45b1a07f0809c5b7dc33d8aa61bc9d1eb0e0f35c79c558a446e2bbfca6e422: Status 404 returned error can't find the container with id 0c45b1a07f0809c5b7dc33d8aa61bc9d1eb0e0f35c79c558a446e2bbfca6e422 Oct 31 01:47:54.990002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.989968 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:47:54.990095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.990035 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:47:54.990095 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:54.990063 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:47:55.788432 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:55.788393 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" event={"ID":"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad","Type":"ContainerStarted","Data":"27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717"} Oct 31 01:47:55.788850 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:55.788457 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" event={"ID":"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad","Type":"ContainerStarted","Data":"0c45b1a07f0809c5b7dc33d8aa61bc9d1eb0e0f35c79c558a446e2bbfca6e422"} Oct 31 01:47:55.955418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:55.955351 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podStartSLOduration=1.9553355350000001 podStartE2EDuration="1.955335535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:55.953368977 +0000 UTC m=+3047.760336521" watchObservedRunningTime="2026-10-31 01:47:55.955335535 +0000 UTC m=+3047.762303077" Oct 31 01:47:56.520964 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:56.520923 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:47:56.522577 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:56.522557 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 01:47:56.522704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:56.522601 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 01:47:57.521968 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:57.521930 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 01:47:57.522372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:57.521986 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 01:47:57.857909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:57.857777 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf"] Oct 31 01:47:58.521526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:58.521491 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 01:47:58.521708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:58.521541 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 01:47:59.521186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:59.521155 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 01:47:59.521684 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:47:59.521212 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 01:48:00.522031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:00.521994 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 01:48:00.522471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:00.522054 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 01:48:01.521464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:01.521430 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 01:48:01.521654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:01.521495 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 01:48:02.066597 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:48:02.066555 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:48:02.525833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:02.525764 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:48:02.526206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:02.526175 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:48:02.526796 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:02.526782 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:48:02.858734 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:02.858696 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" containerName="payload-processing" containerID="cri-o://51bfd5b3092bf50d205f7cba9e08e3da4c9b890afc9536fcd35b7492b0f12845" gracePeriod=25 Oct 31 01:48:05.307783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:05.307741 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:48:06.666940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:06.666905 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:48:08.551591 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.551559 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj"] Oct 31 01:48:08.555081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.555060 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.603428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.603398 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.603608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.603437 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkgw5\" (UniqueName: \"kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.603725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.603699 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.612154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.612122 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj"] Oct 31 01:48:08.656649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.656616 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:48:08.656814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.656736 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:48:08.704919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.704885 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.704919 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.704928 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkgw5\" (UniqueName: \"kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.705138 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.705068 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.705633 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.705611 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.707658 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.707633 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.761475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.761450 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkgw5\" (UniqueName: \"kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5\") pod \"maas-api-revoke-sub-35de0857-57hhj\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:08.868810 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:08.868739 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:09.242599 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:09.242566 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj"] Oct 31 01:48:09.244007 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:48:09.243974 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cb53d4_5fb7_4fa2_b67d_6aa93e4f019c.slice/crio-2821c58f65a356db013ee515d9ed27f97bfa83ee241f15ef7f457b393656399c WatchSource:0}: Error finding container 2821c58f65a356db013ee515d9ed27f97bfa83ee241f15ef7f457b393656399c: Status 404 returned error can't find the container with id 2821c58f65a356db013ee515d9ed27f97bfa83ee241f15ef7f457b393656399c Oct 31 01:48:09.898183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:09.898096 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" event={"ID":"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c","Type":"ContainerStarted","Data":"9a3caaf546ad9d341dff9c78345a9577839acc5c84aa296cc4f697e382a7f4d0"} Oct 31 01:48:09.898183 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:09.898153 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" event={"ID":"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c","Type":"ContainerStarted","Data":"2821c58f65a356db013ee515d9ed27f97bfa83ee241f15ef7f457b393656399c"} Oct 31 01:48:10.010737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.010682 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" podStartSLOduration=2.010668906 podStartE2EDuration="2.010668906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:10.007364495 +0000 UTC m=+3061.814332038" watchObservedRunningTime="2026-10-31 01:48:10.010668906 +0000 UTC m=+3061.817636449" Oct 31 01:48:10.325232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.325203 5756 scope.go:122] "RemoveContainer" containerID="5fca679070277536b1d9bf6778c73bad3d40bb622107ea6be10b95214c14594a" Oct 31 01:48:10.646754 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.646713 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7"] Oct 31 01:48:10.650462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.650444 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.730590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.730544 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.730777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.730720 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.730923 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.730907 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5w9\" (UniqueName: \"kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.745483 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.745459 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7"] Oct 31 01:48:10.832381 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.832348 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5w9\" (UniqueName: \"kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.832532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.832410 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.832532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.832473 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.833174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.833149 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.834938 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.834921 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.888902 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.888848 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5w9\" (UniqueName: \"kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9\") pod \"maas-api-revoke-sub-ee86d873-tmtf7\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:10.906568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.906500 5756 generic.go:356] "Generic (PLEG): container finished" podID="94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" containerID="9a3caaf546ad9d341dff9c78345a9577839acc5c84aa296cc4f697e382a7f4d0" exitCode=0 Oct 31 01:48:10.906568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.906543 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" event={"ID":"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c","Type":"ContainerDied","Data":"9a3caaf546ad9d341dff9c78345a9577839acc5c84aa296cc4f697e382a7f4d0"} Oct 31 01:48:10.962936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:10.962909 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:11.339196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:11.339160 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7"] Oct 31 01:48:11.339646 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:48:11.339613 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf966bec_6f32_479a_b08d_ef843498a13e.slice/crio-57680685352e8bb44f0e3c69be83a7d725da258a53883291f2d6b4473fc0cff2 WatchSource:0}: Error finding container 57680685352e8bb44f0e3c69be83a7d725da258a53883291f2d6b4473fc0cff2: Status 404 returned error can't find the container with id 57680685352e8bb44f0e3c69be83a7d725da258a53883291f2d6b4473fc0cff2 Oct 31 01:48:11.914486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:11.914397 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" event={"ID":"cf966bec-6f32-479a-b08d-ef843498a13e","Type":"ContainerStarted","Data":"badce32d94cd0b01a41c9137bd55d8f6f28943d04348db234bf801cf08900649"} Oct 31 01:48:11.914486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:11.914450 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" event={"ID":"cf966bec-6f32-479a-b08d-ef843498a13e","Type":"ContainerStarted","Data":"57680685352e8bb44f0e3c69be83a7d725da258a53883291f2d6b4473fc0cff2"} Oct 31 01:48:12.031163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.031103 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" podStartSLOduration=2.031086295 podStartE2EDuration="2.031086295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:12.026250689 +0000 UTC m=+3063.833218232" watchObservedRunningTime="2026-10-31 01:48:12.031086295 +0000 UTC m=+3063.838053839" Oct 31 01:48:12.068494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.068471 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:12.149162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.149131 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca\") pod \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " Oct 31 01:48:12.149328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.149252 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5\" (UniqueName: \"kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5\") pod \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " Oct 31 01:48:12.149328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.149292 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca\") pod \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\" (UID: \"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c\") " Oct 31 01:48:12.149481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.149460 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca" pod "94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" (UID: "94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:12.149740 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.149716 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:12.151463 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.151443 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5" pod "94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" (UID: "94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c"). InnerVolumeSpecName "kube-api-access-fkgw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:12.151559 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.151484 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca" pod "94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" (UID: "94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:12.250814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.250736 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkgw5\" (UniqueName: \"kubernetes.io/projected/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-kube-api-access-fkgw5\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:12.250814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.250764 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:12.922393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.922367 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" Oct 31 01:48:12.922808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.922357 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj" event={"ID":"94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c","Type":"ContainerDied","Data":"2821c58f65a356db013ee515d9ed27f97bfa83ee241f15ef7f457b393656399c"} Oct 31 01:48:12.922808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.922487 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2821c58f65a356db013ee515d9ed27f97bfa83ee241f15ef7f457b393656399c" Oct 31 01:48:12.924103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.924078 5756 generic.go:356] "Generic (PLEG): container finished" podID="cf966bec-6f32-479a-b08d-ef843498a13e" containerID="badce32d94cd0b01a41c9137bd55d8f6f28943d04348db234bf801cf08900649" exitCode=0 Oct 31 01:48:12.924236 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:12.924108 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" event={"ID":"cf966bec-6f32-479a-b08d-ef843498a13e","Type":"ContainerDied","Data":"badce32d94cd0b01a41c9137bd55d8f6f28943d04348db234bf801cf08900649"} Oct 31 01:48:14.088627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.088603 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:14.173340 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.173281 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca\") pod \"cf966bec-6f32-479a-b08d-ef843498a13e\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " Oct 31 01:48:14.173560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.173458 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9\" (UniqueName: \"kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9\") pod \"cf966bec-6f32-479a-b08d-ef843498a13e\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " Oct 31 01:48:14.173560 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.173517 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca\") pod \"cf966bec-6f32-479a-b08d-ef843498a13e\" (UID: \"cf966bec-6f32-479a-b08d-ef843498a13e\") " Oct 31 01:48:14.173690 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.173665 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca" pod "cf966bec-6f32-479a-b08d-ef843498a13e" (UID: "cf966bec-6f32-479a-b08d-ef843498a13e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:14.174017 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.173991 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:14.176002 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.175975 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9" pod "cf966bec-6f32-479a-b08d-ef843498a13e" (UID: "cf966bec-6f32-479a-b08d-ef843498a13e"). InnerVolumeSpecName "kube-api-access-7w5w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:14.176165 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.176095 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca" pod "cf966bec-6f32-479a-b08d-ef843498a13e" (UID: "cf966bec-6f32-479a-b08d-ef843498a13e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:14.275077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.275047 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5w9\" (UniqueName: \"kubernetes.io/projected/cf966bec-6f32-479a-b08d-ef843498a13e-kube-api-access-7w5w9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:14.275077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.275074 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf966bec-6f32-479a-b08d-ef843498a13e-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:14.940025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.939998 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" Oct 31 01:48:14.940206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.940027 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7" event={"ID":"cf966bec-6f32-479a-b08d-ef843498a13e","Type":"ContainerDied","Data":"57680685352e8bb44f0e3c69be83a7d725da258a53883291f2d6b4473fc0cff2"} Oct 31 01:48:14.940206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:14.940101 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57680685352e8bb44f0e3c69be83a7d725da258a53883291f2d6b4473fc0cff2" Oct 31 01:48:15.307336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:15.307294 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 01:48:22.999597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:22.999567 5756 generic.go:356] "Generic (PLEG): container finished" podID="51432c69-09be-4042-9f50-dae704e0cd2c" containerID="51bfd5b3092bf50d205f7cba9e08e3da4c9b890afc9536fcd35b7492b0f12845" exitCode=0 Oct 31 01:48:23.000009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:22.999612 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" event={"ID":"51432c69-09be-4042-9f50-dae704e0cd2c","Type":"ContainerDied","Data":"51bfd5b3092bf50d205f7cba9e08e3da4c9b890afc9536fcd35b7492b0f12845"} Oct 31 01:48:23.128460 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.128402 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:48:23.277186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.277150 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc\" (UniqueName: \"kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc\") pod \"51432c69-09be-4042-9f50-dae704e0cd2c\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " Oct 31 01:48:23.277409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.277392 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume\") pod \"51432c69-09be-4042-9f50-dae704e0cd2c\" (UID: \"51432c69-09be-4042-9f50-dae704e0cd2c\") " Oct 31 01:48:23.277817 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.277787 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume" pod "51432c69-09be-4042-9f50-dae704e0cd2c" (UID: "51432c69-09be-4042-9f50-dae704e0cd2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:23.279916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.279897 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc" pod "51432c69-09be-4042-9f50-dae704e0cd2c" (UID: "51432c69-09be-4042-9f50-dae704e0cd2c"). InnerVolumeSpecName "kube-api-access-m42tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:23.379502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.379424 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51432c69-09be-4042-9f50-dae704e0cd2c-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:23.379502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:23.379450 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42tc\" (UniqueName: \"kubernetes.io/projected/51432c69-09be-4042-9f50-dae704e0cd2c-kube-api-access-m42tc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:48:24.009182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:24.009155 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" Oct 31 01:48:24.009604 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:24.009155 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf" event={"ID":"51432c69-09be-4042-9f50-dae704e0cd2c","Type":"ContainerDied","Data":"2607f7814f5ae9efbbd011cfdf58c8b926fc26aaa364096d30d4a47d0dc311c9"} Oct 31 01:48:24.009604 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:24.009287 5756 scope.go:122] "RemoveContainer" containerID="51bfd5b3092bf50d205f7cba9e08e3da4c9b890afc9536fcd35b7492b0f12845" Oct 31 01:48:24.175379 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:24.175344 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf"] Oct 31 01:48:24.182161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:24.182134 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-25b9ff-6866c4fdff-jrgkf"] Oct 31 01:48:24.311722 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:24.311690 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" path="/var/lib/kubelet/pods/51432c69-09be-4042-9f50-dae704e0cd2c/volumes" Oct 31 01:48:32.068110 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:48:32.068072 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:48:33.533587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.533551 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b"] Oct 31 01:48:33.534114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534099 5756 cpu_manager.go:414] "containerMap: removing container" podUID="94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" containerName="revoke-keys" Oct 31 01:48:33.534166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534115 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" containerName="revoke-keys" Oct 31 01:48:33.534166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534138 5756 cpu_manager.go:414] "containerMap: removing container" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" containerName="payload-processing" Oct 31 01:48:33.534166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534143 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" containerName="payload-processing" Oct 31 01:48:33.534327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534173 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cf966bec-6f32-479a-b08d-ef843498a13e" containerName="revoke-keys" Oct 31 01:48:33.534327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534178 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf966bec-6f32-479a-b08d-ef843498a13e" containerName="revoke-keys" Oct 31 01:48:33.534327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534256 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf966bec-6f32-479a-b08d-ef843498a13e" containerName="revoke-keys" Oct 31 01:48:33.534327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534266 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" containerName="revoke-keys" Oct 31 01:48:33.534327 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.534275 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="51432c69-09be-4042-9f50-dae704e0cd2c" containerName="payload-processing" Oct 31 01:48:33.540286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.540256 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.592998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.592953 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-b6f24b-openshift-default-dockercfg-p4p8h\"" Oct 31 01:48:33.593405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593373 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593441 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593524 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593504 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593614 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593575 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593615 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593769 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593752 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593795 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85qbc\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.593838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.593826 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.594031 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.594012 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.646999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.646963 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b"] Oct 31 01:48:33.695264 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695236 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695274 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85qbc\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695296 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695433 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695554 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695625 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695600 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695636 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695681 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695688 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695772 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.695973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.695878 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.696081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.696022 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.696139 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.696081 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.696422 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.696399 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.697956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.697935 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.698126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.698110 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.752482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.752449 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.752736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.752711 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85qbc\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc\") pod \"e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:33.856622 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:33.856536 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:34.301807 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:34.301776 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b"] Oct 31 01:48:34.302904 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:48:34.302850 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d7c3bf_fe2d_4829_ae99_0b5d3746d3a0.slice/crio-51dee869a0c21932dec036d1a75c9810f13aec8d289c285cc818dad35c75e63a WatchSource:0}: Error finding container 51dee869a0c21932dec036d1a75c9810f13aec8d289c285cc818dad35c75e63a: Status 404 returned error can't find the container with id 51dee869a0c21932dec036d1a75c9810f13aec8d289c285cc818dad35c75e63a Oct 31 01:48:34.305949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:34.305914 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:48:34.306061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:34.305994 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:48:34.306061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:34.306040 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:48:35.088967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:35.088933 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" event={"ID":"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0","Type":"ContainerStarted","Data":"4aa14ed803d05d5225ab160830f17c1cde122e021903bb6c9f86ed059ea00180"} Oct 31 01:48:35.089433 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:35.088984 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" event={"ID":"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0","Type":"ContainerStarted","Data":"51dee869a0c21932dec036d1a75c9810f13aec8d289c285cc818dad35c75e63a"} Oct 31 01:48:35.254234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:35.254179 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podStartSLOduration=2.254164014 podStartE2EDuration="2.254164014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:35.253324424 +0000 UTC m=+3087.060291980" watchObservedRunningTime="2026-10-31 01:48:35.254164014 +0000 UTC m=+3087.061131567" Oct 31 01:48:35.856890 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:35.856834 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:35.862649 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:35.862622 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:36.079960 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:36.079921 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c"] Oct 31 01:48:36.083028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:36.083008 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8244c7e5-7rd4c"] Oct 31 01:48:36.095133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:36.095104 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:36.096084 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:36.096064 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:48:36.311332 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:36.311293 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3169a51-7385-4060-ab3d-6e8e920f2d61" path="/var/lib/kubelet/pods/d3169a51-7385-4060-ab3d-6e8e920f2d61/volumes" Oct 31 01:48:43.804884 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.804689 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp"] Oct 31 01:48:43.814423 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.814390 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:43.866762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.866730 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-b6f24b\"" Oct 31 01:48:43.870328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.870305 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp"] Oct 31 01:48:43.914454 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.914427 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-b6f24b-dockercfg-nfw7h\"" Oct 31 01:48:43.932384 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.932359 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htntk\" (UniqueName: \"kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk\") pod \"payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:43.932536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:43.932526 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:44.033737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.033705 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:44.033952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.033810 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htntk\" (UniqueName: \"kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk\") pod \"payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:44.034476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.034453 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:44.047490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.047459 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj"] Oct 31 01:48:44.050510 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.050490 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7ccbea-5trwj"] Oct 31 01:48:44.090219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.090159 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htntk\" (UniqueName: \"kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk\") pod \"payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:44.130636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.130608 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:44.311731 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.311701 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65e10483-22b9-48b8-9d7d-111458d3cc37" path="/var/lib/kubelet/pods/65e10483-22b9-48b8-9d7d-111458d3cc37/volumes" Oct 31 01:48:44.509497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:44.509469 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp"] Oct 31 01:48:44.511092 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:48:44.511065 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298b64a8_a86b_4988_bfb5_bd0467920df2.slice/crio-1f2a95b758e76c5aec8138ed314d2ea823d3eff6afed0b8363a9760c5482e3b0 WatchSource:0}: Error finding container 1f2a95b758e76c5aec8138ed314d2ea823d3eff6afed0b8363a9760c5482e3b0: Status 404 returned error can't find the container with id 1f2a95b758e76c5aec8138ed314d2ea823d3eff6afed0b8363a9760c5482e3b0 Oct 31 01:48:45.166295 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:45.166255 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" event={"ID":"298b64a8-a86b-4988-bfb5-bd0467920df2","Type":"ContainerStarted","Data":"312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9"} Oct 31 01:48:45.166701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:45.166327 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:48:45.166701 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:45.166345 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" event={"ID":"298b64a8-a86b-4988-bfb5-bd0467920df2","Type":"ContainerStarted","Data":"1f2a95b758e76c5aec8138ed314d2ea823d3eff6afed0b8363a9760c5482e3b0"} Oct 31 01:48:45.280612 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:45.280559 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" podStartSLOduration=2.280542804 podStartE2EDuration="2.280542804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:45.279495236 +0000 UTC m=+3097.086462780" watchObservedRunningTime="2026-10-31 01:48:45.280542804 +0000 UTC m=+3097.087510346" Oct 31 01:48:52.047161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:52.047130 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx"] Oct 31 01:48:52.096166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:52.096134 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4f83914-2qxnx"] Oct 31 01:48:52.311062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:52.310984 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bd79ba-3d07-4b8c-9991-da320d0d68c7" path="/var/lib/kubelet/pods/d0bd79ba-3d07-4b8c-9991-da320d0d68c7/volumes" Oct 31 01:48:56.180385 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:48:56.180359 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 01:49:00.051406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:00.051374 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8"] Oct 31 01:49:00.100232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:00.100198 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fd8bea5-vkqz8"] Oct 31 01:49:00.311003 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:00.310925 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b376785-4691-42e6-a35d-14bb5de40dd5" path="/var/lib/kubelet/pods/4b376785-4691-42e6-a35d-14bb5de40dd5/volumes" Oct 31 01:49:01.802703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.802671 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6"] Oct 31 01:49:01.810388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.810358 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:01.863166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.863133 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6"] Oct 31 01:49:01.908147 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.908082 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7fc058\"" Oct 31 01:49:01.917967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.916786 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:01.918275 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.918254 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nq9w\" (UniqueName: \"kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w\") pod \"payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:01.929190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:01.929155 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7fc058-dockercfg-jvzm7\"" Oct 31 01:49:02.019606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:02.019568 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:02.019814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:02.019625 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nq9w\" (UniqueName: \"kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w\") pod \"payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:02.020508 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:02.020483 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:02.074195 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:49:02.074111 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:49:02.075906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:02.075885 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nq9w\" (UniqueName: \"kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w\") pod \"payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:02.125499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:02.125467 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:02.499082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:02.499050 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6"] Oct 31 01:49:02.499902 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:49:02.499843 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18978975_2d04_4264_859a_d9288ae28e25.slice/crio-5852fe8700ad9b00776c8fb2250a56b21ae37d26e93b1ae6e7b0ac1050a7cbdc WatchSource:0}: Error finding container 5852fe8700ad9b00776c8fb2250a56b21ae37d26e93b1ae6e7b0ac1050a7cbdc: Status 404 returned error can't find the container with id 5852fe8700ad9b00776c8fb2250a56b21ae37d26e93b1ae6e7b0ac1050a7cbdc Oct 31 01:49:03.317449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:03.317419 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" event={"ID":"18978975-2d04-4264-859a-d9288ae28e25","Type":"ContainerStarted","Data":"bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f"} Oct 31 01:49:03.317892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:03.317462 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" event={"ID":"18978975-2d04-4264-859a-d9288ae28e25","Type":"ContainerStarted","Data":"5852fe8700ad9b00776c8fb2250a56b21ae37d26e93b1ae6e7b0ac1050a7cbdc"} Oct 31 01:49:03.317892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:03.317636 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:03.449370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:03.449315 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" podStartSLOduration=2.449299316 podStartE2EDuration="2.449299316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:03.44765367 +0000 UTC m=+3115.254621212" watchObservedRunningTime="2026-10-31 01:49:03.449299316 +0000 UTC m=+3115.256266859" Oct 31 01:49:05.079568 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:05.079534 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c"] Oct 31 01:49:05.128770 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:05.128738 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-25b9ff-7j75c"] Oct 31 01:49:06.200250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:06.200218 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:49:06.311437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:06.311406 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce286ca-b87f-41dc-aa65-efa45a5f139a" path="/var/lib/kubelet/pods/2ce286ca-b87f-41dc-aa65-efa45a5f139a/volumes" Oct 31 01:49:09.049532 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:09.049498 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp"] Oct 31 01:49:09.098617 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:09.098584 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-775be981-nzvsp"] Oct 31 01:49:10.311108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:10.311072 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8436aee-129f-4df4-86b4-9754971c7fd0" path="/var/lib/kubelet/pods/f8436aee-129f-4df4-86b4-9754971c7fd0/volumes" Oct 31 01:49:10.363272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:10.363239 5756 scope.go:122] "RemoveContainer" containerID="392a0c04b79c1b53af92ce0cd3b090b17fd0bdfed68f7adbe15915395680b849" Oct 31 01:49:10.373781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:10.373758 5756 scope.go:122] "RemoveContainer" containerID="de3f4490a1cb38f6cc30ea9cf98d4af5f36d19cfcf00893a21a7596e6a98f98b" Oct 31 01:49:14.331207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:14.331178 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 01:49:17.048455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:17.048419 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj"] Oct 31 01:49:17.097431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:17.097391 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76e42945-7f8mj"] Oct 31 01:49:18.314904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:18.314850 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1545929-62cc-4e1a-82b4-3371baf04b1d" path="/var/lib/kubelet/pods/d1545929-62cc-4e1a-82b4-3371baf04b1d/volumes" Oct 31 01:49:20.045723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:20.045693 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr"] Oct 31 01:49:20.066337 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:20.066308 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3658b19a-6mcqr"] Oct 31 01:49:20.313182 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:20.313101 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166a6c2a-c77b-4c09-8a8e-600c47dc2ddc" path="/var/lib/kubelet/pods/166a6c2a-c77b-4c09-8a8e-600c47dc2ddc/volumes" Oct 31 01:49:26.705939 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.705902 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76"] Oct 31 01:49:26.711652 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.711632 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:26.771993 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.771958 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76"] Oct 31 01:49:26.816641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.816607 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-9c9c3c\"" Oct 31 01:49:26.816808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.816648 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-9c9c3c-dockercfg-h65mk\"" Oct 31 01:49:26.826161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.826139 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdtqf\" (UniqueName: \"kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf\") pod \"payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:26.826364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.826349 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:26.927272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.927238 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:26.927464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.927296 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdtqf\" (UniqueName: \"kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf\") pod \"payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:26.927977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.927958 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:26.984988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:26.984915 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdtqf\" (UniqueName: \"kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf\") pod \"payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:27.026407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:27.026377 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:27.402506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:27.402480 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76"] Oct 31 01:49:27.403144 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:49:27.403115 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f1a671_4208_4d65_9eff_d9ba3be97be0.slice/crio-b45a56af7ee56006b6eb38b5d2205e2eddc5dca763fa6101209832ece17cbd8c WatchSource:0}: Error finding container b45a56af7ee56006b6eb38b5d2205e2eddc5dca763fa6101209832ece17cbd8c: Status 404 returned error can't find the container with id b45a56af7ee56006b6eb38b5d2205e2eddc5dca763fa6101209832ece17cbd8c Oct 31 01:49:27.490662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:27.490625 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" event={"ID":"81f1a671-4208-4d65-9eff-d9ba3be97be0","Type":"ContainerStarted","Data":"9ab2e62451c8245aa4a75c6b8d2bf452abc10e31acd9c539232d898169c4bf59"} Oct 31 01:49:27.490816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:27.490690 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:27.490816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:27.490708 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" event={"ID":"81f1a671-4208-4d65-9eff-d9ba3be97be0","Type":"ContainerStarted","Data":"b45a56af7ee56006b6eb38b5d2205e2eddc5dca763fa6101209832ece17cbd8c"} Oct 31 01:49:27.606089 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:27.606033 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" podStartSLOduration=1.60602069 podStartE2EDuration="1.60602069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:27.604809843 +0000 UTC m=+3139.411777383" watchObservedRunningTime="2026-10-31 01:49:27.60602069 +0000 UTC m=+3139.412988232" Oct 31 01:49:28.054277 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:28.054241 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz"] Oct 31 01:49:28.057171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:28.057145 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-317e77d9-mftzz"] Oct 31 01:49:28.311536 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:28.311449 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48594c70-93a6-4a03-99b1-c58fa948883f" path="/var/lib/kubelet/pods/48594c70-93a6-4a03-99b1-c58fa948883f/volumes" Oct 31 01:49:32.066135 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:49:32.066095 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:49:37.049745 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:37.049711 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc"] Oct 31 01:49:37.099291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:37.099262 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-767c655c-dx4xc"] Oct 31 01:49:38.313168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:38.313136 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a96ade5d-922b-4e3a-9ae7-d435ca0e0a72" path="/var/lib/kubelet/pods/a96ade5d-922b-4e3a-9ae7-d435ca0e0a72/volumes" Oct 31 01:49:38.501216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:38.501188 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 01:49:44.048208 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:44.048172 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9"] Oct 31 01:49:44.097290 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:44.097250 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70fa5b34-27tc9"] Oct 31 01:49:44.312642 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:44.312561 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c118b5-48ff-432a-b8a9-dc52e06979f4" path="/var/lib/kubelet/pods/24c118b5-48ff-432a-b8a9-dc52e06979f4/volumes" Oct 31 01:49:55.115086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.115014 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh"] Oct 31 01:49:55.120371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.120352 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.174184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.174124 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh"] Oct 31 01:49:55.219654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.219616 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:49:55.219823 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.219713 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:49:55.286720 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.286685 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmd86\" (UniqueName: \"kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.286955 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.286847 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.286955 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.286949 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.388700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.388612 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.388700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.388672 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.388999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.388806 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmd86\" (UniqueName: \"kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.389359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.389334 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.391141 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.391123 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.447302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.447277 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmd86\" (UniqueName: \"kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86\") pod \"maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:55.733755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:55.733679 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:56.130574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:56.130533 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh"] Oct 31 01:49:56.132070 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:49:56.132035 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cf8e0d_5196_4a19_a6bc_445e601f044f.slice/crio-cef6756cde0553e5c3b5267fcfc0e79ffbfbd238425c1be647834855271750e1 WatchSource:0}: Error finding container cef6756cde0553e5c3b5267fcfc0e79ffbfbd238425c1be647834855271750e1: Status 404 returned error can't find the container with id cef6756cde0553e5c3b5267fcfc0e79ffbfbd238425c1be647834855271750e1 Oct 31 01:49:56.702985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:56.702892 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" event={"ID":"76cf8e0d-5196-4a19-a6bc-445e601f044f","Type":"ContainerStarted","Data":"360d6d44e0896640f48e6bb5d0aeb6865420e9dccda225bc75e95d5a616d10ef"} Oct 31 01:49:56.702985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:56.702956 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" event={"ID":"76cf8e0d-5196-4a19-a6bc-445e601f044f","Type":"ContainerStarted","Data":"cef6756cde0553e5c3b5267fcfc0e79ffbfbd238425c1be647834855271750e1"} Oct 31 01:49:56.816490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:56.816426 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" podStartSLOduration=1.816411132 podStartE2EDuration="1.816411132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:56.8133982 +0000 UTC m=+3168.620365756" watchObservedRunningTime="2026-10-31 01:49:56.816411132 +0000 UTC m=+3168.623378675" Oct 31 01:49:57.361318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:57.361274 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:49:57.711733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:57.711642 5756 generic.go:356] "Generic (PLEG): container finished" podID="76cf8e0d-5196-4a19-a6bc-445e601f044f" containerID="360d6d44e0896640f48e6bb5d0aeb6865420e9dccda225bc75e95d5a616d10ef" exitCode=0 Oct 31 01:49:57.711921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:57.711734 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" event={"ID":"76cf8e0d-5196-4a19-a6bc-445e601f044f","Type":"ContainerDied","Data":"360d6d44e0896640f48e6bb5d0aeb6865420e9dccda225bc75e95d5a616d10ef"} Oct 31 01:49:58.923146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:58.923118 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:59.058730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.058681 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca\") pod \"76cf8e0d-5196-4a19-a6bc-445e601f044f\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " Oct 31 01:49:59.059126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.059085 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86\" (UniqueName: \"kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86\") pod \"76cf8e0d-5196-4a19-a6bc-445e601f044f\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " Oct 31 01:49:59.060020 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.059856 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca\") pod \"76cf8e0d-5196-4a19-a6bc-445e601f044f\" (UID: \"76cf8e0d-5196-4a19-a6bc-445e601f044f\") " Oct 31 01:49:59.060270 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.060210 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca" pod "76cf8e0d-5196-4a19-a6bc-445e601f044f" (UID: "76cf8e0d-5196-4a19-a6bc-445e601f044f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:49:59.060814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.060786 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:49:59.061709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.061687 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca" pod "76cf8e0d-5196-4a19-a6bc-445e601f044f" (UID: "76cf8e0d-5196-4a19-a6bc-445e601f044f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:49:59.067629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.067603 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86" pod "76cf8e0d-5196-4a19-a6bc-445e601f044f" (UID: "76cf8e0d-5196-4a19-a6bc-445e601f044f"). InnerVolumeSpecName "kube-api-access-tmd86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:49:59.162258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.162222 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76cf8e0d-5196-4a19-a6bc-445e601f044f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:49:59.162258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.162252 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmd86\" (UniqueName: \"kubernetes.io/projected/76cf8e0d-5196-4a19-a6bc-445e601f044f-kube-api-access-tmd86\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:49:59.727294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.727263 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" Oct 31 01:49:59.727294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.727288 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh" event={"ID":"76cf8e0d-5196-4a19-a6bc-445e601f044f","Type":"ContainerDied","Data":"cef6756cde0553e5c3b5267fcfc0e79ffbfbd238425c1be647834855271750e1"} Oct 31 01:49:59.727523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:49:59.727336 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cef6756cde0553e5c3b5267fcfc0e79ffbfbd238425c1be647834855271750e1" Oct 31 01:50:02.092886 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:50:02.092807 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:50:05.876779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:05.876746 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74"] Oct 31 01:50:07.137027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.136988 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:50:07.137459 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.137046 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:50:07.807176 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.807144 5756 generic.go:356] "Generic (PLEG): container finished" podID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerID="4aa14ed803d05d5225ab160830f17c1cde122e021903bb6c9f86ed059ea00180" exitCode=0 Oct 31 01:50:07.807389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.807224 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" event={"ID":"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0","Type":"ContainerDied","Data":"4aa14ed803d05d5225ab160830f17c1cde122e021903bb6c9f86ed059ea00180"} Oct 31 01:50:07.807681 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.807662 5756 scope.go:122] "RemoveContainer" containerID="4aa14ed803d05d5225ab160830f17c1cde122e021903bb6c9f86ed059ea00180" Oct 31 01:50:07.809227 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.809198 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:50:07.809313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.809271 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:50:07.809313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:07.809305 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:50:08.816528 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:08.816487 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" event={"ID":"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0","Type":"ContainerStarted","Data":"f934569aae395a604139df8d0348336ab4e41cbaa8e21b8da8c3a7d8f2cfd35c"} Oct 31 01:50:08.857389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:08.857345 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:50:08.857592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:08.857494 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:50:08.857592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:08.857555 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 01:50:09.857879 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:09.857826 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:50:09.858355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:09.857930 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 01:50:10.411062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.411035 5756 scope.go:122] "RemoveContainer" containerID="1e02fe11391adb2f1764efca30a970ce9418bf6435ed16862b9cd5f13b42278b" Oct 31 01:50:10.422909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.422884 5756 scope.go:122] "RemoveContainer" containerID="0d227546f57c6fa00891483f80e37fa0d8fc2ae35ce5a5a3dd68f75bda9c8e48" Oct 31 01:50:10.460065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.460040 5756 scope.go:122] "RemoveContainer" containerID="44a2f4c68e5ba1d200cb7a46a439886818016f2b8c6fc3749bf8e6ac025e07e2" Oct 31 01:50:10.469691 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.469670 5756 scope.go:122] "RemoveContainer" containerID="d85709a42ea80dd479e4c778403f40535d1a45a5f9ea99fabbd2c602f3516937" Oct 31 01:50:10.479080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.479059 5756 scope.go:122] "RemoveContainer" containerID="96b719d028730a578c1516234c71921c1fe7aec8b511fc9d8bcd06fec59f51cb" Oct 31 01:50:10.856889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.856841 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:50:10.857111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.856923 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 01:50:10.877293 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:10.877246 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" podUID="0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" containerName="payload-processing" containerID="cri-o://6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb" gracePeriod=25 Oct 31 01:50:11.186174 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.186139 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:50:11.306034 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.306006 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume\") pod \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " Oct 31 01:50:11.306285 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.306257 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn\" (UniqueName: \"kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn\") pod \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\" (UID: \"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b\") " Oct 31 01:50:11.306339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.306322 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume" pod "0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" (UID: "0bd2808f-9e8e-44d2-ab6e-3eaab892a35b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:50:11.306743 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.306718 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:50:11.308471 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.308448 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn" pod "0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" (UID: "0bd2808f-9e8e-44d2-ab6e-3eaab892a35b"). InnerVolumeSpecName "kube-api-access-ztbsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:50:11.407925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.407891 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbsn\" (UniqueName: \"kubernetes.io/projected/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b-kube-api-access-ztbsn\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:50:11.857093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.857044 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:50:11.857302 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.857099 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 01:50:11.861061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.861031 5756 generic.go:356] "Generic (PLEG): container finished" podID="0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" containerID="6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb" exitCode=0 Oct 31 01:50:11.861224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.861077 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" Oct 31 01:50:11.861224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.861100 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" event={"ID":"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b","Type":"ContainerDied","Data":"6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb"} Oct 31 01:50:11.861224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.861170 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74" event={"ID":"0bd2808f-9e8e-44d2-ab6e-3eaab892a35b","Type":"ContainerDied","Data":"e02e9a8868f94d0027d1a702b06a97814652085b3116868005e605c7863ae3c0"} Oct 31 01:50:11.861224 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.861197 5756 scope.go:122] "RemoveContainer" containerID="6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb" Oct 31 01:50:11.873710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.873691 5756 scope.go:122] "RemoveContainer" containerID="6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb" Oct 31 01:50:11.873996 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:50:11.873977 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb\": container with ID starting with 6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb not found: ID does not exist" containerID="6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb" Oct 31 01:50:11.874053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:11.874005 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb"} err="failed to get container status \"6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb\": rpc error: code = NotFound desc = could not find container \"6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb\": container with ID starting with 6b6507ed6e408b392dc592416bffa5792221792aa04ade72626fe6fe3b26efbb not found: ID does not exist" Oct 31 01:50:12.042950 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:12.042913 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74"] Oct 31 01:50:12.054894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:12.053293 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ade881-559866dbc9-wck74"] Oct 31 01:50:12.311228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:12.311196 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" path="/var/lib/kubelet/pods/0bd2808f-9e8e-44d2-ab6e-3eaab892a35b/volumes" Oct 31 01:50:12.857558 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:12.857526 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:50:12.857748 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:12.857597 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 01:50:13.857533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:13.857496 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 01:50:13.858061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:13.857554 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 01:50:14.861321 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:14.861292 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:50:14.861811 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:14.861716 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:50:14.862383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:14.862362 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 01:50:19.051643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:19.051611 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m"] Oct 31 01:50:19.101061 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:19.101030 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c22e2353-v725m"] Oct 31 01:50:20.312398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:20.312368 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3c7885-9bc0-4daa-8381-0fb2f83b9831" path="/var/lib/kubelet/pods/0b3c7885-9bc0-4daa-8381-0fb2f83b9831/volumes" Oct 31 01:50:25.970636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:25.970593 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:50:32.066242 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:50:32.066202 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:50:34.048688 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:34.048634 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2"] Oct 31 01:50:34.067494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:34.067465 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79452fa3-xwxl2"] Oct 31 01:50:34.311700 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:34.311626 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabf28dd-c287-46bd-9cb3-18ab9c564221" path="/var/lib/kubelet/pods/eabf28dd-c287-46bd-9cb3-18ab9c564221/volumes" Oct 31 01:50:37.051675 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:37.051644 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6"] Oct 31 01:50:37.101073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:37.101040 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e4fb4d-v2gh6"] Oct 31 01:50:38.311546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:38.311513 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9fb9eb3-c456-43ff-be56-186b3469dd4d" path="/var/lib/kubelet/pods/b9fb9eb3-c456-43ff-be56-186b3469dd4d/volumes" Oct 31 01:50:44.246998 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:44.246962 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh"] Oct 31 01:50:44.296371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:44.296331 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ade881-ww2nh"] Oct 31 01:50:44.312073 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:44.312044 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cf8e0d-5196-4a19-a6bc-445e601f044f" path="/var/lib/kubelet/pods/76cf8e0d-5196-4a19-a6bc-445e601f044f/volumes" Oct 31 01:50:47.175170 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.175134 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6"] Oct 31 01:50:47.176590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.176562 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" containerName="payload-processing" Oct 31 01:50:47.176590 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.176590 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" containerName="payload-processing" Oct 31 01:50:47.176809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.176625 5756 cpu_manager.go:414] "containerMap: removing container" podUID="76cf8e0d-5196-4a19-a6bc-445e601f044f" containerName="revoke-keys" Oct 31 01:50:47.176809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.176635 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cf8e0d-5196-4a19-a6bc-445e601f044f" containerName="revoke-keys" Oct 31 01:50:47.176809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.176754 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd2808f-9e8e-44d2-ab6e-3eaab892a35b" containerName="payload-processing" Oct 31 01:50:47.176809 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.176771 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cf8e0d-5196-4a19-a6bc-445e601f044f" containerName="revoke-keys" Oct 31 01:50:47.180220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.180200 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.273921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.272383 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-95404c-openshift-default-dockercfg-4q9g7\"" Oct 31 01:50:47.310414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.310378 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6"] Oct 31 01:50:47.311702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.311666 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.311827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.311727 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsls\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.311886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.311833 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.311999 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.311983 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.312078 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.312025 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.312167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.312077 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.312167 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.312159 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.312294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.312207 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.312294 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.312238 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413271 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413322 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413360 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413387 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsls\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413468 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413500 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413526 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413564 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.413629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.413618 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.414048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.414022 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.414373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.414351 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.414788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.414647 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.414788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.414718 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.414788 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.414748 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.417181 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.417131 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.418207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.418186 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.471606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.471527 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsls\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.471991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.471967 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token\") pod \"e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.496534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.496501 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:47.940836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.940814 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6"] Oct 31 01:50:47.942500 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:50:47.942472 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f8af91_50d7_495a_a080_aa71bf3b68ed.slice/crio-89ec910f3a7c26ec47e1a9218f4b5a6deab1709b6d9f178ef2bcaae2d955f96b WatchSource:0}: Error finding container 89ec910f3a7c26ec47e1a9218f4b5a6deab1709b6d9f178ef2bcaae2d955f96b: Status 404 returned error can't find the container with id 89ec910f3a7c26ec47e1a9218f4b5a6deab1709b6d9f178ef2bcaae2d955f96b Oct 31 01:50:47.945571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.945531 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:50:47.945673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.945607 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:50:47.945673 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:47.945637 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:50:48.140889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:48.140558 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" event={"ID":"53f8af91-50d7-495a-a080-aa71bf3b68ed","Type":"ContainerStarted","Data":"7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7"} Oct 31 01:50:48.140889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:48.140628 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" event={"ID":"53f8af91-50d7-495a-a080-aa71bf3b68ed","Type":"ContainerStarted","Data":"89ec910f3a7c26ec47e1a9218f4b5a6deab1709b6d9f178ef2bcaae2d955f96b"} Oct 31 01:50:48.288616 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:48.288549 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podStartSLOduration=1.288531762 podStartE2EDuration="1.288531762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:48.284564374 +0000 UTC m=+3220.091531923" watchObservedRunningTime="2026-10-31 01:50:48.288531762 +0000 UTC m=+3220.095499306" Oct 31 01:50:48.498559 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:48.498520 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:49.048941 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:49.048910 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp"] Oct 31 01:50:49.097924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:49.097892 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f771f4c-xmdwp"] Oct 31 01:50:49.504067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:49.503992 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:50.156733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:50.156701 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:50.157808 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:50.157785 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:50:50.312882 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:50.312831 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648064f7-8cbb-4231-a83c-e68ac986dd80" path="/var/lib/kubelet/pods/648064f7-8cbb-4231-a83c-e68ac986dd80/volumes" Oct 31 01:50:55.343910 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.343841 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl"] Oct 31 01:50:55.349719 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.349694 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.404996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.404963 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl"] Oct 31 01:50:55.410086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.409469 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-95404c\"" Oct 31 01:50:55.410086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.409578 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-95404c-dockercfg-kqckw\"" Oct 31 01:50:55.430276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.430242 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4jl\" (UniqueName: \"kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl\") pod \"payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.430519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.430481 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.532913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.532856 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4jl\" (UniqueName: \"kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl\") pod \"payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.533107 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.533040 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.533782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.533761 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.593406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.593371 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4jl\" (UniqueName: \"kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl\") pod \"payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:55.663204 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:55.663116 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:56.046511 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:56.046488 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl"] Oct 31 01:50:56.047470 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:50:56.047445 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e49c4a_7ffd_44ad_9e6b_685f9ae30821.slice/crio-dbc864a1882c355fb0d7e9beef9cc0cbef06e93ab6772f4e1ed10f44275fccd2 WatchSource:0}: Error finding container dbc864a1882c355fb0d7e9beef9cc0cbef06e93ab6772f4e1ed10f44275fccd2: Status 404 returned error can't find the container with id dbc864a1882c355fb0d7e9beef9cc0cbef06e93ab6772f4e1ed10f44275fccd2 Oct 31 01:50:56.197450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:56.197412 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" event={"ID":"17e49c4a-7ffd-44ad-9e6b-685f9ae30821","Type":"ContainerStarted","Data":"f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce"} Oct 31 01:50:56.197638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:56.197495 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" event={"ID":"17e49c4a-7ffd-44ad-9e6b-685f9ae30821","Type":"ContainerStarted","Data":"dbc864a1882c355fb0d7e9beef9cc0cbef06e93ab6772f4e1ed10f44275fccd2"} Oct 31 01:50:56.197638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:56.197542 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:50:56.314464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:50:56.314364 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" podStartSLOduration=1.314351297 podStartE2EDuration="1.314351297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:56.312418868 +0000 UTC m=+3228.119386412" watchObservedRunningTime="2026-10-31 01:50:56.314351297 +0000 UTC m=+3228.121318864" Oct 31 01:51:02.066652 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:51:02.066613 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:51:07.207389 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:07.207358 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 01:51:10.525880 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:10.525841 5756 scope.go:122] "RemoveContainer" containerID="4395fb3ff514c97d85633bc19b78855f243150e50efef0ba2efe5fd69b123d78" Oct 31 01:51:10.536573 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:10.536547 5756 scope.go:122] "RemoveContainer" containerID="58df49b8cd5a004de41c52eaa8feb4042c1a3cfcc358f20ce769c953bef9885a" Oct 31 01:51:10.548650 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:10.548624 5756 scope.go:122] "RemoveContainer" containerID="507a2a0a27ea154e49297385c348817977dccef85d1e3524c3d417ca9a09a967" Oct 31 01:51:10.557957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:10.557935 5756 scope.go:122] "RemoveContainer" containerID="9b828371b7238baa820ddeb3e97fd5544120453ce4805a7ab87a2b0fcb546bca" Oct 31 01:51:10.566839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:10.566821 5756 scope.go:122] "RemoveContainer" containerID="4549eaa0f7ee99546f64108808d8db6ac2b16644754797b08e7c5d14f842d9cc" Oct 31 01:51:10.575449 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:10.575426 5756 scope.go:122] "RemoveContainer" containerID="588c36e6ac006aee83abbf47832050d4cbc298ef7426239682cfadc94fecff8d" Oct 31 01:51:20.054046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:20.053963 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp"] Oct 31 01:51:20.102372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:20.102336 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9295b6c5-8gcbp"] Oct 31 01:51:20.259830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:20.259790 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:51:20.311470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:20.311394 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd6993e5-4f4b-42e8-b119-19c1c88bc46e" path="/var/lib/kubelet/pods/fd6993e5-4f4b-42e8-b119-19c1c88bc46e/volumes" Oct 31 01:51:22.047841 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:22.047796 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n"] Oct 31 01:51:22.096987 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:22.096959 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade59137-4zv2n"] Oct 31 01:51:22.311839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:22.311765 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809e120e-134e-4ff4-9513-b0037c01066d" path="/var/lib/kubelet/pods/809e120e-134e-4ff4-9513-b0037c01066d/volumes" Oct 31 01:51:32.067037 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:51:32.067002 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:51:34.048641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:34.048607 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp"] Oct 31 01:51:34.097834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:34.097793 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-794c9efd-tj5tp"] Oct 31 01:51:34.311103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:34.311022 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="918af203-6923-4d19-8514-4e59bd945ae5" path="/var/lib/kubelet/pods/918af203-6923-4d19-8514-4e59bd945ae5/volumes" Oct 31 01:51:45.048388 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:45.048354 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp"] Oct 31 01:51:45.105885 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:45.099162 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3936f42e-nf8qp"] Oct 31 01:51:46.311782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:46.311743 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="039d23d7-fe0b-4ab0-9cd6-845b76a3853c" path="/var/lib/kubelet/pods/039d23d7-fe0b-4ab0-9cd6-845b76a3853c/volumes" Oct 31 01:51:49.047163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:49.047131 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q"] Oct 31 01:51:49.098234 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:49.098201 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac870d2-k7g5q"] Oct 31 01:51:50.311526 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:50.311491 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3177ea16-91bb-41ba-85ca-f51ce70dd71a" path="/var/lib/kubelet/pods/3177ea16-91bb-41ba-85ca-f51ce70dd71a/volumes" Oct 31 01:51:52.047448 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:52.047411 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl"] Oct 31 01:51:52.096708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:52.096676 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12c611-2vbnl"] Oct 31 01:51:52.313437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:51:52.313359 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1984ee3a-c2a5-4f5c-861c-d30ac881cf86" path="/var/lib/kubelet/pods/1984ee3a-c2a5-4f5c-861c-d30ac881cf86/volumes" Oct 31 01:52:01.203049 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.203009 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r"] Oct 31 01:52:01.206575 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.206556 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.257708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.257674 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.257708 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.257711 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.257943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.257826 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn67v\" (UniqueName: \"kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.257943 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.257876 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.258013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.257967 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.258013 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.258002 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.310152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.310117 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b992fe\"/\"kube-root-ca.crt\"" Oct 31 01:52:01.310152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.310149 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b992fe\"/\"openshift-service-ca.crt\"" Oct 31 01:52:01.310398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.310180 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b992fe\"/\"default-dockercfg-xn5gl\"" Oct 31 01:52:01.310398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.310219 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b992fe\"/\"auth-test-model-b992fe-kserve-self-signed-certs\"" Oct 31 01:52:01.312870 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.312818 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r"] Oct 31 01:52:01.359350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359314 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359350 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359431 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn67v\" (UniqueName: \"kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359464 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359512 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359702 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359559 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359899 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.359977 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359914 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.360083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.359993 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.361715 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.361693 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.362179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.362158 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.417193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.417162 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn67v\" (UniqueName: \"kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v\") pod \"auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.517818 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.517731 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:01.904565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:01.904539 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r"] Oct 31 01:52:01.905943 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:52:01.905917 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b89314f_373c_47c6_b499_0abfcb749301.slice/crio-abeab07903abc1f59698ab56310637f222fd1dc13b0ca525bd221046e25729cd WatchSource:0}: Error finding container abeab07903abc1f59698ab56310637f222fd1dc13b0ca525bd221046e25729cd: Status 404 returned error can't find the container with id abeab07903abc1f59698ab56310637f222fd1dc13b0ca525bd221046e25729cd Oct 31 01:52:02.051888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:02.051845 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n"] Oct 31 01:52:02.054408 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:02.054379 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dae6b7d-gk65n"] Oct 31 01:52:02.066959 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:52:02.066929 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:52:02.312007 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:02.311967 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715d5105-69c6-4587-be86-3404d1ca113c" path="/var/lib/kubelet/pods/715d5105-69c6-4587-be86-3404d1ca113c/volumes" Oct 31 01:52:02.661256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:02.661162 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" event={"ID":"9b89314f-373c-47c6-b499-0abfcb749301","Type":"ContainerStarted","Data":"743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a"} Oct 31 01:52:02.661256 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:02.661209 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" event={"ID":"9b89314f-373c-47c6-b499-0abfcb749301","Type":"ContainerStarted","Data":"abeab07903abc1f59698ab56310637f222fd1dc13b0ca525bd221046e25729cd"} Oct 31 01:52:02.784733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:02.784675 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" podStartSLOduration=1.784658899 podStartE2EDuration="1.784658899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:02.783239518 +0000 UTC m=+3294.590207060" watchObservedRunningTime="2026-10-31 01:52:02.784658899 +0000 UTC m=+3294.591626441" Oct 31 01:52:03.517916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:03.517887 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:10.627421 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:10.627391 5756 scope.go:122] "RemoveContainer" containerID="a1b87ab0d8350c7ad594b33431b6838929e59266ca4fb51345f103044264d11e" Oct 31 01:52:10.637535 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:10.637511 5756 scope.go:122] "RemoveContainer" containerID="9f15bf7e4db9dd02700d435d55aed9a5f5480239493859261938aeed5ab39ff0" Oct 31 01:52:10.646594 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:10.646578 5756 scope.go:122] "RemoveContainer" containerID="81201cc6d5b31879503021568dd53958fb05bdf0ec86fbe41cdeeb0ee840abed" Oct 31 01:52:10.655569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:10.655552 5756 scope.go:122] "RemoveContainer" containerID="beea736867df7b4e8a1d647a2dd0f20efe4df7fd8383de2c83741f3a3a641b0d" Oct 31 01:52:10.664051 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:10.664036 5756 scope.go:122] "RemoveContainer" containerID="163655be0316a383d83dfe5981c7ff05e7ff258b7a4a8a90024cb79296cd713e" Oct 31 01:52:11.518710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:11.518668 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:11.531553 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:11.531526 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:11.738991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:11.738958 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:52:17.402069 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:17.402034 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:52:28.865370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:28.865331 5756 generic.go:356] "Generic (PLEG): container finished" podID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerID="7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7" exitCode=0 Oct 31 01:52:28.865926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:28.865401 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" event={"ID":"53f8af91-50d7-495a-a080-aa71bf3b68ed","Type":"ContainerDied","Data":"7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7"} Oct 31 01:52:28.865926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:28.865845 5756 scope.go:122] "RemoveContainer" containerID="7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7" Oct 31 01:52:28.867490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:28.867461 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:52:28.867631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:28.867540 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:52:28.867631 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:28.867589 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:52:29.873309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:29.873270 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" event={"ID":"53f8af91-50d7-495a-a080-aa71bf3b68ed","Type":"ContainerStarted","Data":"722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af"} Oct 31 01:52:30.498286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:30.498251 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:52:30.498529 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:30.498506 5756 patch_prober.go:29] interesting pod/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 01:52:30.498627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:30.498573 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 01:52:31.499081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:31.499049 5756 patch_prober.go:29] interesting pod/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 01:52:31.499479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:31.499103 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 01:52:32.078336 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:52:32.078302 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:52:32.498781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:32.498698 5756 patch_prober.go:29] interesting pod/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 01:52:32.498781 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:32.498751 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 01:52:33.498442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:33.498404 5756 patch_prober.go:29] interesting pod/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 01:52:33.498831 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:33.498471 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 01:52:34.498940 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:34.498901 5756 patch_prober.go:29] interesting pod/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 31 01:52:34.499339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:34.498972 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 31 01:52:35.156645 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:35.156610 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:52:35.503491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:35.503417 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:52:35.917315 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:35.917288 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 01:52:46.632312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:46.632232 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:52:56.592124 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.592086 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc"] Oct 31 01:52:56.592672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.592301 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" containerID="cri-o://44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a" gracePeriod=30 Oct 31 01:52:56.642028 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.641995 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn"] Oct 31 01:52:56.642272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.642223 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" containerID="cri-o://27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717" gracePeriod=30 Oct 31 01:52:56.644238 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.644217 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn"] Oct 31 01:52:56.644409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.644389 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" containerID="cri-o://65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a" gracePeriod=30 Oct 31 01:52:56.691065 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.691031 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx"] Oct 31 01:52:56.691276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:52:56.691253 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" containerID="cri-o://29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635" gracePeriod=30 Oct 31 01:53:01.526009 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:01.525974 5756 patch_prober.go:29] interesting pod/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:53:01.526455 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:01.526036 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:53:01.938936 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:01.938908 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:53:02.006744 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.006715 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:53:02.047795 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.047738 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:53:02.052015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.051983 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:53:02.052356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052339 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.052450 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052422 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.052553 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052533 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.052619 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052603 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.052662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052635 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.052662 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052657 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.052768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052679 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.052768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052681 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.052768 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052707 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.052974 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052783 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.052974 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052848 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.052974 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052904 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.052974 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052934 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.052982 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053040 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053052 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053085 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053118 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053145 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.053206 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053176 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053210 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053250 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053291 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053325 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053373 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053421 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053454 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053513 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053538 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053569 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token\") pod \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\" (UID: \"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053618 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.053771 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053656 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.054521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.054148 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.054521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.054163 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.054521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.054173 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.054521 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.053291 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.054751 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.054562 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.055557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.054937 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.055557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.055109 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:02.055557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.055239 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.055909 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.055815 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056050 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056123 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056156 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056209 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056246 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056531 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056642 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.056832 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.057077 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.057152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.057131 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.058361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.058310 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.058361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.058328 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.058361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.058342 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.060080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.059980 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "kube-api-access-vngdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.060080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.060013 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "kube-api-access-hqqh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.060080 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.060056 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:53:02.060409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.060362 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:53:02.060409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.060370 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "kube-api-access-s9ck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.060409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.060386 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "kube-api-access-hvmvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.061046 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.061028 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy" pod "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" (UID: "dbb1af39-ea05-4b0b-8a83-90bb2257fcf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.072094 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:02.072059 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:53:02.092563 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.092536 5756 generic.go:356] "Generic (PLEG): container finished" podID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerID="27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717" exitCode=0 Oct 31 01:53:02.092750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.092584 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" Oct 31 01:53:02.092750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.092613 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" event={"ID":"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad","Type":"ContainerDied","Data":"27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717"} Oct 31 01:53:02.092750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.092672 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn" event={"ID":"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad","Type":"ContainerDied","Data":"0c45b1a07f0809c5b7dc33d8aa61bc9d1eb0e0f35c79c558a446e2bbfca6e422"} Oct 31 01:53:02.092750 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.092706 5756 scope.go:122] "RemoveContainer" containerID="27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717" Oct 31 01:53:02.094005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.093988 5756 generic.go:356] "Generic (PLEG): container finished" podID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerID="65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a" exitCode=0 Oct 31 01:53:02.094105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.094030 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" Oct 31 01:53:02.094105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.094055 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" event={"ID":"924bc7ad-57f1-4d79-a666-ce618a60ca4c","Type":"ContainerDied","Data":"65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a"} Oct 31 01:53:02.094105 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.094092 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn" event={"ID":"924bc7ad-57f1-4d79-a666-ce618a60ca4c","Type":"ContainerDied","Data":"43e82d61cfb8f3001631a6e9f810644c0a18b6765442d807858d512049e69d1f"} Oct 31 01:53:02.095834 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.095813 5756 generic.go:356] "Generic (PLEG): container finished" podID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerID="44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a" exitCode=0 Oct 31 01:53:02.095963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.095903 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" Oct 31 01:53:02.096039 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.095961 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" event={"ID":"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0","Type":"ContainerDied","Data":"44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a"} Oct 31 01:53:02.096039 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.096004 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc" event={"ID":"dbb1af39-ea05-4b0b-8a83-90bb2257fcf0","Type":"ContainerDied","Data":"e64cd841541200059cce86ecc86ce09c8903e11bf272554753606511ec02095a"} Oct 31 01:53:02.097365 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.097345 5756 generic.go:356] "Generic (PLEG): container finished" podID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerID="29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635" exitCode=0 Oct 31 01:53:02.097482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.097387 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" event={"ID":"092f149c-9821-422c-8d26-aafeb22ac5ea","Type":"ContainerDied","Data":"29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635"} Oct 31 01:53:02.097482 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.097404 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" Oct 31 01:53:02.097606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.097418 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx" event={"ID":"092f149c-9821-422c-8d26-aafeb22ac5ea","Type":"ContainerDied","Data":"94e964407eb515606a77c9d364647ff3f4ca92f71b306a10c235ed9d9e0b7d26"} Oct 31 01:53:02.103660 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.103633 5756 scope.go:122] "RemoveContainer" containerID="27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717" Oct 31 01:53:02.104760 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:02.104738 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717\": container with ID starting with 27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717 not found: ID does not exist" containerID="27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717" Oct 31 01:53:02.104900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.104769 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717"} err="failed to get container status \"27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717\": rpc error: code = NotFound desc = could not find container \"27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717\": container with ID starting with 27c21c38286b1c74eabc2399409c30bad77154c332778892f254dd68b727b717 not found: ID does not exist" Oct 31 01:53:02.104900 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.104795 5756 scope.go:122] "RemoveContainer" containerID="65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a" Oct 31 01:53:02.116603 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.116584 5756 scope.go:122] "RemoveContainer" containerID="65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a" Oct 31 01:53:02.117060 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:02.117036 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a\": container with ID starting with 65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a not found: ID does not exist" containerID="65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a" Oct 31 01:53:02.117178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.117071 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a"} err="failed to get container status \"65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a\": rpc error: code = NotFound desc = could not find container \"65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a\": container with ID starting with 65d15ebfc8d0a6acbecfb0d914eda2f6a2b6ee142aec582b64ab81a1019d372a not found: ID does not exist" Oct 31 01:53:02.117178 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.117096 5756 scope.go:122] "RemoveContainer" containerID="44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a" Oct 31 01:53:02.130816 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.130794 5756 scope.go:122] "RemoveContainer" containerID="d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872" Oct 31 01:53:02.142410 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.142387 5756 scope.go:122] "RemoveContainer" containerID="44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a" Oct 31 01:53:02.142763 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:02.142745 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a\": container with ID starting with 44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a not found: ID does not exist" containerID="44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a" Oct 31 01:53:02.142835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.142781 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a"} err="failed to get container status \"44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a\": rpc error: code = NotFound desc = could not find container \"44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a\": container with ID starting with 44fb6410f700716082f91c699e65bcd8440e487fe28b059ebef8cbb3fb6bb53a not found: ID does not exist" Oct 31 01:53:02.142835 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.142801 5756 scope.go:122] "RemoveContainer" containerID="d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872" Oct 31 01:53:02.143105 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:02.143086 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872\": container with ID starting with d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872 not found: ID does not exist" containerID="d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872" Oct 31 01:53:02.143156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.143112 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872"} err="failed to get container status \"d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872\": rpc error: code = NotFound desc = could not find container \"d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872\": container with ID starting with d3da6c479222af68c23698ae760b25f7606e77d212bfb0ac00afd563f4a61872 not found: ID does not exist" Oct 31 01:53:02.143156 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.143131 5756 scope.go:122] "RemoveContainer" containerID="29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635" Oct 31 01:53:02.152724 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.152704 5756 scope.go:122] "RemoveContainer" containerID="29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635" Oct 31 01:53:02.153076 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:02.153050 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635\": container with ID starting with 29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635 not found: ID does not exist" containerID="29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635" Oct 31 01:53:02.153163 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.153086 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635"} err="failed to get container status \"29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635\": rpc error: code = NotFound desc = could not find container \"29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635\": container with ID starting with 29cba19d9f604c8515713d2ea5a77b5cdc4ac33ec7295543a97aa770cfe5c635 not found: ID does not exist" Oct 31 01:53:02.155587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155566 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.155674 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155601 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket\") pod \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\" (UID: \"924bc7ad-57f1-4d79-a666-ce618a60ca4c\") " Oct 31 01:53:02.155779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155760 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.155821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155779 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.155821 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155805 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.155984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155839 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token\") pod \"092f149c-9821-422c-8d26-aafeb22ac5ea\" (UID: \"092f149c-9821-422c-8d26-aafeb22ac5ea\") " Oct 31 01:53:02.155984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155906 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.155984 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.155936 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket\") pod \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\" (UID: \"8eeb0e8f-51a3-4608-a56a-f50e7ba325ad\") " Oct 31 01:53:02.156159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156042 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.156220 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156202 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.156284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156215 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data" pod "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" (UID: "8eeb0e8f-51a3-4608-a56a-f50e7ba325ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.156475 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156457 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156476 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156486 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156495 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156507 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156520 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156534 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156543 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156551 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156561 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156570 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156578 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156587 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156595 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9ck6\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-kube-api-access-s9ck6\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156603 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156611 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156619 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.156623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156627 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vngdc\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-kube-api-access-vngdc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156637 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156645 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156653 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156661 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156669 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqh5\" (UniqueName: \"kubernetes.io/projected/924bc7ad-57f1-4d79-a666-ce618a60ca4c-kube-api-access-hqqh5\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156677 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156685 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092f149c-9821-422c-8d26-aafeb22ac5ea-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156694 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156702 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156710 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156718 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvmvs\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-kube-api-access-hvmvs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156727 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/924bc7ad-57f1-4d79-a666-ce618a60ca4c-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.157172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.156772 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:02.157833 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.157819 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo" pod "924bc7ad-57f1-4d79-a666-ce618a60ca4c" (UID: "924bc7ad-57f1-4d79-a666-ce618a60ca4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:53:02.158106 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.158088 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token" pod "092f149c-9821-422c-8d26-aafeb22ac5ea" (UID: "092f149c-9821-422c-8d26-aafeb22ac5ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:02.257544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.257509 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/924bc7ad-57f1-4d79-a666-ce618a60ca4c-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.257544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.257541 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092f149c-9821-422c-8d26-aafeb22ac5ea-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.257544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.257553 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092f149c-9821-422c-8d26-aafeb22ac5ea-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:53:02.383499 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.383471 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc"] Oct 31 01:53:02.389933 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.389906 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-n5gqc"] Oct 31 01:53:02.729407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.729378 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn"] Oct 31 01:53:02.734287 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.734261 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-wxlmn"] Oct 31 01:53:02.842971 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.842932 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx"] Oct 31 01:53:02.847111 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.847085 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-qbpkx"] Oct 31 01:53:02.955268 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.955190 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn"] Oct 31 01:53:02.959502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:02.959475 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e621db-openshift-default-5c44f894cc-g4rvn"] Oct 31 01:53:04.311153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:04.311121 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" path="/var/lib/kubelet/pods/092f149c-9821-422c-8d26-aafeb22ac5ea/volumes" Oct 31 01:53:04.311564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:04.311551 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" path="/var/lib/kubelet/pods/8eeb0e8f-51a3-4608-a56a-f50e7ba325ad/volumes" Oct 31 01:53:04.311988 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:04.311967 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" path="/var/lib/kubelet/pods/924bc7ad-57f1-4d79-a666-ce618a60ca4c/volumes" Oct 31 01:53:04.312607 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:04.312590 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" path="/var/lib/kubelet/pods/dbb1af39-ea05-4b0b-8a83-90bb2257fcf0/volumes" Oct 31 01:53:08.331205 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331175 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr"] Oct 31 01:53:08.331668 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331655 5756 cpu_manager.go:414] "containerMap: removing container" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331669 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331691 5756 cpu_manager.go:414] "containerMap: removing container" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331698 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331708 5756 cpu_manager.go:414] "containerMap: removing container" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331713 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331723 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" Oct 31 01:53:08.331726 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331728 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331737 5756 cpu_manager.go:414] "containerMap: removing container" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331742 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331844 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="092f149c-9821-422c-8d26-aafeb22ac5ea" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331875 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="924bc7ad-57f1-4d79-a666-ce618a60ca4c" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331885 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331894 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eeb0e8f-51a3-4608-a56a-f50e7ba325ad" containerName="istio-proxy" Oct 31 01:53:08.332038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.331904 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb1af39-ea05-4b0b-8a83-90bb2257fcf0" containerName="istio-proxy" Oct 31 01:53:08.335394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.335367 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.433439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.433403 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.433627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.433519 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.433627 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.433592 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjnf\" (UniqueName: \"kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.433763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.433738 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.433840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.433823 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.433921 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.433908 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.440896 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.440847 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-083e17\"/\"auth-test-model-083e17-kserve-self-signed-certs\"" Oct 31 01:53:08.440995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.440925 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-083e17\"/\"default-dockercfg-gh2q7\"" Oct 31 01:53:08.440995 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.440960 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-083e17\"/\"openshift-service-ca.crt\"" Oct 31 01:53:08.441192 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.441171 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-083e17\"/\"kube-root-ca.crt\"" Oct 31 01:53:08.449240 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.449219 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr"] Oct 31 01:53:08.535162 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535132 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535178 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535198 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535281 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535324 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535371 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535346 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjnf\" (UniqueName: \"kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535578 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.535737 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.535718 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.536063 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.536038 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.537992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.537969 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.538100 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.537998 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.599884 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.599789 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjnf\" (UniqueName: \"kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf\") pod \"auth-test-model-083e17-kserve-6c56744bfc-jmxdr\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:08.649447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:08.649358 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:09.056115 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:09.056090 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr"] Oct 31 01:53:09.057903 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:53:09.057858 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05a3a1e2_dfd8_4d4d_aa27_7cb9b502324a.slice/crio-44e07770b34650bb9c002f047bdb27c9432d881c3ef9d8415bd69b65ac66d571 WatchSource:0}: Error finding container 44e07770b34650bb9c002f047bdb27c9432d881c3ef9d8415bd69b65ac66d571: Status 404 returned error can't find the container with id 44e07770b34650bb9c002f047bdb27c9432d881c3ef9d8415bd69b65ac66d571 Oct 31 01:53:09.156304 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:09.156269 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" event={"ID":"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a","Type":"ContainerStarted","Data":"2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52"} Oct 31 01:53:09.156476 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:09.156313 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" event={"ID":"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a","Type":"ContainerStarted","Data":"44e07770b34650bb9c002f047bdb27c9432d881c3ef9d8415bd69b65ac66d571"} Oct 31 01:53:09.276932 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:09.276883 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" podStartSLOduration=1.276849928 podStartE2EDuration="1.276849928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:09.275996615 +0000 UTC m=+3361.082964159" watchObservedRunningTime="2026-10-31 01:53:09.276849928 +0000 UTC m=+3361.083817472" Oct 31 01:53:09.649820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:09.649788 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:10.709053 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.709024 5756 scope.go:122] "RemoveContainer" containerID="b2c81eb27a2e03e583a8dc5f8f286d3711a2737e2cf2ecfc2054c8ddce39a582" Oct 31 01:53:10.721839 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.721812 5756 scope.go:122] "RemoveContainer" containerID="27c3f497fa0a202f7ef4859d75e2ed4218152044e12a8cc6537523bf0b0688b0" Oct 31 01:53:10.730813 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.730791 5756 scope.go:122] "RemoveContainer" containerID="44facac5c627f0f6c733dcc1d9d9ec09e27f020dbd5bb3275ed2cd7a3dcb4de2" Oct 31 01:53:10.739555 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.739534 5756 scope.go:122] "RemoveContainer" containerID="44d08612a3740b5487ddb7294cbae2e5f8350670d7b6acb4206e64e1baf8faca" Oct 31 01:53:10.748310 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.748291 5756 scope.go:122] "RemoveContainer" containerID="5d4ef9069ef22a12720be6d30d0f94a121ef840326f9de2f5e29c0701465bceb" Oct 31 01:53:10.756830 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.756812 5756 scope.go:122] "RemoveContainer" containerID="2930526b9b1b6c63ed952a667e05edd6f6785ff9b1b833ba91e6ac271523ed1e" Oct 31 01:53:10.765274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.765254 5756 scope.go:122] "RemoveContainer" containerID="2380fb24bc4cba58915ebc08c321574dd6f15eccc0b305c021e8e9402088c771" Oct 31 01:53:10.773646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.773628 5756 scope.go:122] "RemoveContainer" containerID="200f99a0a4d77c6d1dff59e748148f973f51b4dc123a06187f3b9099a6ad91b3" Oct 31 01:53:10.782098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:10.782075 5756 scope.go:122] "RemoveContainer" containerID="fbb970336af33b98484cc88caf8bea7e309ce02e20b1d9ec4e87b303bc857e48" Oct 31 01:53:13.050623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:13.050588 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj"] Oct 31 01:53:13.100149 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:13.100111 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35de0857-57hhj"] Oct 31 01:53:14.311626 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:14.311596 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c" path="/var/lib/kubelet/pods/94cb53d4-5fb7-4fa2-b67d-6aa93e4f019c/volumes" Oct 31 01:53:15.043766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:15.043725 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7"] Oct 31 01:53:15.093286 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:15.093247 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee86d873-tmtf7"] Oct 31 01:53:16.312125 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:16.312092 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf966bec-6f32-479a-b08d-ef843498a13e" path="/var/lib/kubelet/pods/cf966bec-6f32-479a-b08d-ef843498a13e/volumes" Oct 31 01:53:18.649550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:18.649517 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:18.662045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:18.662021 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:19.234347 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:19.234309 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:53:25.126762 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:25.126730 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:53:28.670520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.670485 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd"] Oct 31 01:53:28.677427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.677402 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.733437 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.733398 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd"] Oct 31 01:53:28.779872 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.779836 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fc058\"/\"openshift-service-ca.crt\"" Oct 31 01:53:28.779872 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.779836 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fc058\"/\"kube-root-ca.crt\"" Oct 31 01:53:28.780098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.779910 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fc058\"/\"auth-test-model-7fc058-kserve-self-signed-certs\"" Oct 31 01:53:28.780098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.779951 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fc058\"/\"default-dockercfg-9kvxf\"" Oct 31 01:53:28.803225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.803184 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.803417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.803323 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.803417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.803360 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.803417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.803387 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.803611 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.803588 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6ts\" (UniqueName: \"kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.803690 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.803665 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.904632 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.904593 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.904831 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.904681 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6ts\" (UniqueName: \"kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.904831 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.904750 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.904949 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.904845 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.905019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.904979 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.905086 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.905024 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.905393 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.905369 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.905507 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.905487 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.905638 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.905613 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.907407 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.907383 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.907597 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.907578 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:28.965396 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:28.964963 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6ts\" (UniqueName: \"kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts\") pod \"auth-test-model-7fc058-kserve-57b496f65-kzpcd\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:29.002814 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:29.002776 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:29.391887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:29.391847 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd"] Oct 31 01:53:29.393808 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:53:29.393782 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3046b0ec_6da0_4e04_a5f3_e9a729c38fcc.slice/crio-4c94632bdc13311c9e4a3a6d020006953d9251c0de0fe9daa669888305f335b1 WatchSource:0}: Error finding container 4c94632bdc13311c9e4a3a6d020006953d9251c0de0fe9daa669888305f335b1: Status 404 returned error can't find the container with id 4c94632bdc13311c9e4a3a6d020006953d9251c0de0fe9daa669888305f335b1 Oct 31 01:53:30.296168 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:30.296133 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" event={"ID":"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc","Type":"ContainerStarted","Data":"a7518d8316ba918b70c5c691ede990cbcc33311fbaa5490d087492012bf9fc1b"} Oct 31 01:53:30.296840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:30.296179 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" event={"ID":"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc","Type":"ContainerStarted","Data":"4c94632bdc13311c9e4a3a6d020006953d9251c0de0fe9daa669888305f335b1"} Oct 31 01:53:30.410480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:30.410424 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" podStartSLOduration=2.4104107040000002 podStartE2EDuration="2.410410704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:30.408787865 +0000 UTC m=+3382.215755408" watchObservedRunningTime="2026-10-31 01:53:30.410410704 +0000 UTC m=+3382.217378247" Oct 31 01:53:31.003057 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:31.003022 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:32.066972 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:53:32.066935 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:53:39.003319 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:39.003282 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:39.016429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:39.016399 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:53:39.372721 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:53:39.372690 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:54:02.067272 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:54:02.067237 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:54:02.705274 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:02.705236 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:54:10.861188 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:10.861156 5756 scope.go:122] "RemoveContainer" containerID="9a3caaf546ad9d341dff9c78345a9577839acc5c84aa296cc4f697e382a7f4d0" Oct 31 01:54:10.871081 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:10.871063 5756 scope.go:122] "RemoveContainer" containerID="981386f0d15119fa968cba9d6848bae78af94675205a8442887bfb34c148eaeb" Oct 31 01:54:29.694394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.694313 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g"] Oct 31 01:54:29.700004 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.699982 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.752904 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.752836 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g"] Oct 31 01:54:29.798942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.798901 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c9c3c\"/\"openshift-service-ca.crt\"" Oct 31 01:54:29.798942 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.798916 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c9c3c\"/\"kube-root-ca.crt\"" Oct 31 01:54:29.799204 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.799011 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c9c3c\"/\"default-dockercfg-jk7dr\"" Oct 31 01:54:29.800894 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.800842 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-9c9c3c\"/\"auth-test-model-9c9c3c-kserve-self-signed-certs\"" Oct 31 01:54:29.859918 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.859879 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.860114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.859997 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.860114 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.860055 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.860225 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.860207 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.860351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.860334 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.860428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.860406 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jwt\" (UniqueName: \"kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961388 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961435 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961557 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961670 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961630 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961738 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961843 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961783 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jwt\" (UniqueName: \"kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961844 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.961951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.961945 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.962145 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.962117 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.964012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.963985 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:29.964468 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:29.964452 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:30.018250 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:30.018217 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jwt\" (UniqueName: \"kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt\") pod \"auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:30.312464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:30.312436 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:30.695324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:30.695293 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g"] Oct 31 01:54:30.697135 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:54:30.697092 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8f121e3_3480_4349_8424_21241105353a.slice/crio-4d3270de768a02917765ffb0e9877e434dab36aa38244523f5ed509b8fe21e30 WatchSource:0}: Error finding container 4d3270de768a02917765ffb0e9877e434dab36aa38244523f5ed509b8fe21e30: Status 404 returned error can't find the container with id 4d3270de768a02917765ffb0e9877e434dab36aa38244523f5ed509b8fe21e30 Oct 31 01:54:31.687166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:31.687125 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" event={"ID":"d8f121e3-3480-4349-8424-21241105353a","Type":"ContainerStarted","Data":"02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7"} Oct 31 01:54:31.687166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:31.687171 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" event={"ID":"d8f121e3-3480-4349-8424-21241105353a","Type":"ContainerStarted","Data":"4d3270de768a02917765ffb0e9877e434dab36aa38244523f5ed509b8fe21e30"} Oct 31 01:54:31.803326 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:31.803275 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" podStartSLOduration=2.80326261 podStartE2EDuration="2.80326261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:31.80200021 +0000 UTC m=+3443.608967768" watchObservedRunningTime="2026-10-31 01:54:31.80326261 +0000 UTC m=+3443.610230152" Oct 31 01:54:32.068005 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:54:32.067972 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:54:32.313520 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:32.313486 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:40.313419 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:40.313389 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:40.325679 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:40.325657 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:40.751179 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:40.751098 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:54:58.812687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:54:58.812650 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:55:02.068637 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:55:02.068602 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:55:08.879609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.879570 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs"] Oct 31 01:55:08.883690 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.883667 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:08.935783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.935753 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:08.935982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.935826 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqt4v\" (UniqueName: \"kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:08.935982 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.935945 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:08.940592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.940557 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:55:08.940837 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.940820 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:55:08.947367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:08.947340 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs"] Oct 31 01:55:09.036857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.036828 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.037054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.036925 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqt4v\" (UniqueName: \"kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.037054 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.036965 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.037623 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.037606 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.039317 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.039299 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.093956 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.093922 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqt4v\" (UniqueName: \"kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v\") pod \"maas-api-revoke-keys-e2e-explicit-c12708-9p5zs\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.198284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.198200 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:09.578021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.577987 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs"] Oct 31 01:55:09.579529 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:55:09.579505 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b60b6fe_6c15_400b_9a6a_6a0bdf79f503.slice/crio-f3ca2a1df28c35dfdff3c668601079bb401c8312b817e471071151760f8c6d4e WatchSource:0}: Error finding container f3ca2a1df28c35dfdff3c668601079bb401c8312b817e471071151760f8c6d4e: Status 404 returned error can't find the container with id f3ca2a1df28c35dfdff3c668601079bb401c8312b817e471071151760f8c6d4e Oct 31 01:55:09.933641 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.933601 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" event={"ID":"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503","Type":"ContainerStarted","Data":"21da9097654c096580abc5983becc58b906609c7ca035f6128bcb280d16d72be"} Oct 31 01:55:09.934082 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:09.933667 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" event={"ID":"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503","Type":"ContainerStarted","Data":"f3ca2a1df28c35dfdff3c668601079bb401c8312b817e471071151760f8c6d4e"} Oct 31 01:55:10.061233 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:10.061184 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" podStartSLOduration=2.06116758 podStartE2EDuration="2.06116758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:10.059145695 +0000 UTC m=+3481.866113237" watchObservedRunningTime="2026-10-31 01:55:10.06116758 +0000 UTC m=+3481.868135126" Oct 31 01:55:10.898710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:10.898678 5756 scope.go:122] "RemoveContainer" containerID="badce32d94cd0b01a41c9137bd55d8f6f28943d04348db234bf801cf08900649" Oct 31 01:55:10.941131 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:10.941095 5756 generic.go:356] "Generic (PLEG): container finished" podID="4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" containerID="21da9097654c096580abc5983becc58b906609c7ca035f6128bcb280d16d72be" exitCode=0 Oct 31 01:55:10.941506 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:10.941177 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" event={"ID":"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503","Type":"ContainerDied","Data":"21da9097654c096580abc5983becc58b906609c7ca035f6128bcb280d16d72be"} Oct 31 01:55:12.104077 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.104053 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:12.177228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.177194 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca\") pod \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " Oct 31 01:55:12.177395 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.177386 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca\") pod \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " Oct 31 01:55:12.177436 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.177426 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v\" (UniqueName: \"kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v\") pod \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\" (UID: \"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503\") " Oct 31 01:55:12.177707 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.177684 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca" pod "4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" (UID: "4b60b6fe-6c15-400b-9a6a-6a0bdf79f503"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:55:12.179441 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.179420 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca" pod "4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" (UID: "4b60b6fe-6c15-400b-9a6a-6a0bdf79f503"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:55:12.179559 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.179545 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v" pod "4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" (UID: "4b60b6fe-6c15-400b-9a6a-6a0bdf79f503"). InnerVolumeSpecName "kube-api-access-xqt4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:55:12.278592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.278514 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:55:12.278592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.278543 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:55:12.278592 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.278553 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqt4v\" (UniqueName: \"kubernetes.io/projected/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503-kube-api-access-xqt4v\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:55:12.954451 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.954417 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" Oct 31 01:55:12.954659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.954447 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs" event={"ID":"4b60b6fe-6c15-400b-9a6a-6a0bdf79f503","Type":"ContainerDied","Data":"f3ca2a1df28c35dfdff3c668601079bb401c8312b817e471071151760f8c6d4e"} Oct 31 01:55:12.954659 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:12.954496 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ca2a1df28c35dfdff3c668601079bb401c8312b817e471071151760f8c6d4e" Oct 31 01:55:32.067854 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:55:32.067808 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:55:55.736654 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:55.736609 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs"] Oct 31 01:55:55.785832 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:55.785797 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-c12708-9p5zs"] Oct 31 01:55:56.312385 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:55:56.312352 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" path="/var/lib/kubelet/pods/4b60b6fe-6c15-400b-9a6a-6a0bdf79f503/volumes" Oct 31 01:56:02.066451 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:56:02.066407 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:56:10.925284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:10.925245 5756 scope.go:122] "RemoveContainer" containerID="360d6d44e0896640f48e6bb5d0aeb6865420e9dccda225bc75e95d5a616d10ef" Oct 31 01:56:20.671951 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:20.671907 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:56:28.091159 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:28.091122 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw"] Oct 31 01:56:28.091820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:28.091352 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" containerID="cri-o://07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5" gracePeriod=30 Oct 31 01:56:28.141534 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:28.141492 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th"] Oct 31 01:56:28.141792 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:28.141747 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" containerID="cri-o://122e9d2e704730873717caec3d67d6b83cc3fa1d7a2f4f89b3a1c712598d01ea" gracePeriod=30 Oct 31 01:56:32.069248 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:56:32.069212 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:56:32.569481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:32.569447 5756 patch_prober.go:29] interesting pod/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:56:32.569651 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:32.569508 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:56:33.438764 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.438738 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:56:33.455728 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.455694 5756 generic.go:356] "Generic (PLEG): container finished" podID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerID="122e9d2e704730873717caec3d67d6b83cc3fa1d7a2f4f89b3a1c712598d01ea" exitCode=0 Oct 31 01:56:33.455937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.455770 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" event={"ID":"8d47ab4c-92ab-4632-a924-aaf53a48de97","Type":"ContainerDied","Data":"122e9d2e704730873717caec3d67d6b83cc3fa1d7a2f4f89b3a1c712598d01ea"} Oct 31 01:56:33.457496 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.457471 5756 generic.go:356] "Generic (PLEG): container finished" podID="c56ff08b-72d8-408f-8f13-008638ef60de" containerID="07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5" exitCode=0 Oct 31 01:56:33.457634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.457505 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" event={"ID":"c56ff08b-72d8-408f-8f13-008638ef60de","Type":"ContainerDied","Data":"07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5"} Oct 31 01:56:33.457634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.457529 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" Oct 31 01:56:33.457634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.457564 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw" event={"ID":"c56ff08b-72d8-408f-8f13-008638ef60de","Type":"ContainerDied","Data":"263ce7cc6b2977684c65ea592f799be0a4dabf84990730acd9700f423f857a29"} Oct 31 01:56:33.457634 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.457603 5756 scope.go:122] "RemoveContainer" containerID="07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5" Oct 31 01:56:33.469970 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.469951 5756 scope.go:122] "RemoveContainer" containerID="07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5" Oct 31 01:56:33.470356 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:56:33.470264 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5\": container with ID starting with 07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5 not found: ID does not exist" containerID="07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5" Oct 31 01:56:33.470356 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.470308 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5"} err="failed to get container status \"07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5\": rpc error: code = NotFound desc = could not find container \"07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5\": container with ID starting with 07f94ce5ca4d35107d42462053878ba5bfeb11a1171d8a6a199f3c779592ece5 not found: ID does not exist" Oct 31 01:56:33.492958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.492930 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:56:33.588489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588451 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.588657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588542 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.588657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588585 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.588657 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588626 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.588804 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588687 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.588804 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588744 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.588804 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588793 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.589021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588848 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.589021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588936 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.589021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.588993 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo\") pod \"c56ff08b-72d8-408f-8f13-008638ef60de\" (UID: \"c56ff08b-72d8-408f-8f13-008638ef60de\") " Oct 31 01:56:33.589021 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589005 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.589248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589106 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.589248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589192 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:56:33.589248 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589230 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.589569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589551 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.589644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589575 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.589644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589589 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.589644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589602 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c56ff08b-72d8-408f-8f13-008638ef60de-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.589644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.589617 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.591014 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.590993 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.591161 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.591141 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "kube-api-access-67srm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:56:33.591417 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.591402 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:56:33.591585 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.591561 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo" pod "c56ff08b-72d8-408f-8f13-008638ef60de" (UID: "c56ff08b-72d8-408f-8f13-008638ef60de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:56:33.690523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690450 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690523 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690500 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690543 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690611 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690686 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690751 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690814 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.690937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690910 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.691083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.690995 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket\") pod \"8d47ab4c-92ab-4632-a924-aaf53a48de97\" (UID: \"8d47ab4c-92ab-4632-a924-aaf53a48de97\") " Oct 31 01:56:33.691083 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691028 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.691452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691351 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.691452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691394 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:56:33.691452 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691417 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.691730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691648 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.691730 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691708 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67srm\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-kube-api-access-67srm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.691822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691731 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c56ff08b-72d8-408f-8f13-008638ef60de-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.691822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691747 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.691822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691763 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c56ff08b-72d8-408f-8f13-008638ef60de-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.691822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691777 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.691822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691791 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c56ff08b-72d8-408f-8f13-008638ef60de-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.691822 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691809 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.692103 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.691823 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d47ab4c-92ab-4632-a924-aaf53a48de97-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.692917 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.692857 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:56:33.693016 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.692950 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:56:33.693247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.693233 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:56:33.693647 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.693626 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc" pod "8d47ab4c-92ab-4632-a924-aaf53a48de97" (UID: "8d47ab4c-92ab-4632-a924-aaf53a48de97"). InnerVolumeSpecName "kube-api-access-vcnbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:56:33.793398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.793355 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.793398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.793393 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.793398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.793405 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnbc\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-kube-api-access-vcnbc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.793653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.793414 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.793653 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.793425 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d47ab4c-92ab-4632-a924-aaf53a48de97-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:56:33.881030 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.880995 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw"] Oct 31 01:56:33.884195 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:33.884166 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-9b6fw"] Oct 31 01:56:34.311671 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:34.311638 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" path="/var/lib/kubelet/pods/c56ff08b-72d8-408f-8f13-008638ef60de/volumes" Oct 31 01:56:34.466272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:34.466182 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" event={"ID":"8d47ab4c-92ab-4632-a924-aaf53a48de97","Type":"ContainerDied","Data":"d374e9756e3cb27761b75f753cf6ada834af353f3edb27c23182fcbc92f8ad95"} Oct 31 01:56:34.466272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:34.466208 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th" Oct 31 01:56:34.466272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:34.466249 5756 scope.go:122] "RemoveContainer" containerID="122e9d2e704730873717caec3d67d6b83cc3fa1d7a2f4f89b3a1c712598d01ea" Oct 31 01:56:34.639384 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:34.639349 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th"] Oct 31 01:56:34.688644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:34.688611 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-97b317-openshift-default-747d7dd9df-cv8th"] Oct 31 01:56:36.312824 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:56:36.312793 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" path="/var/lib/kubelet/pods/8d47ab4c-92ab-4632-a924-aaf53a48de97/volumes" Oct 31 01:57:00.687514 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.687484 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx"] Oct 31 01:57:00.688092 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688076 5756 cpu_manager.go:414] "containerMap: removing container" podUID="4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" containerName="revoke-keys" Oct 31 01:57:00.688146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688094 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" containerName="revoke-keys" Oct 31 01:57:00.688146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688107 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" Oct 31 01:57:00.688146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688112 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" Oct 31 01:57:00.688146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688138 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" Oct 31 01:57:00.688146 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688143 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" Oct 31 01:57:00.688312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688245 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56ff08b-72d8-408f-8f13-008638ef60de" containerName="istio-proxy" Oct 31 01:57:00.688312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688261 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b60b6fe-6c15-400b-9a6a-6a0bdf79f503" containerName="revoke-keys" Oct 31 01:57:00.688312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.688269 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d47ab4c-92ab-4632-a924-aaf53a48de97" containerName="istio-proxy" Oct 31 01:57:00.691757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.691741 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.713312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.713285 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.713470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.713345 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz4t\" (UniqueName: \"kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.713470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.713407 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.746438 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.746398 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx"] Oct 31 01:57:00.792857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.792823 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:57:00.792857 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.792846 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:57:00.814312 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.814284 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.814490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.814344 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz4t\" (UniqueName: \"kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.814490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.814375 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.814991 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.814967 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.816985 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.816965 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:00.871983 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:00.871954 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz4t\" (UniqueName: \"kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t\") pod \"maas-api-revoke-sub-22ab9915-fk7nx\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:01.006198 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:01.006110 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:01.381915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:01.381893 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx"] Oct 31 01:57:01.384126 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:57:01.384096 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a85254_6ae0_4990_894e_9ab7d091e313.slice/crio-0be43bf1ab00b7c2f0595036d57ad607dcf8c704d829750f8dd410cb08df4c30 WatchSource:0}: Error finding container 0be43bf1ab00b7c2f0595036d57ad607dcf8c704d829750f8dd410cb08df4c30: Status 404 returned error can't find the container with id 0be43bf1ab00b7c2f0595036d57ad607dcf8c704d829750f8dd410cb08df4c30 Oct 31 01:57:01.636954 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:01.636886 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" event={"ID":"b7a85254-6ae0-4990-894e-9ab7d091e313","Type":"ContainerStarted","Data":"0be43bf1ab00b7c2f0595036d57ad607dcf8c704d829750f8dd410cb08df4c30"} Oct 31 01:57:02.067164 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:57:02.067128 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:57:02.645070 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:02.645038 5756 generic.go:356] "Generic (PLEG): container finished" podID="b7a85254-6ae0-4990-894e-9ab7d091e313" containerID="fb55fba57926bd3a0748484ec1c8abf86581a9361ebd865ca16dad541e4cf70f" exitCode=0 Oct 31 01:57:02.645260 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:02.645097 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" event={"ID":"b7a85254-6ae0-4990-894e-9ab7d091e313","Type":"ContainerDied","Data":"fb55fba57926bd3a0748484ec1c8abf86581a9361ebd865ca16dad541e4cf70f"} Oct 31 01:57:03.798920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.798896 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:03.856406 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.856373 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca\") pod \"b7a85254-6ae0-4990-894e-9ab7d091e313\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " Oct 31 01:57:03.856595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.856456 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca\") pod \"b7a85254-6ae0-4990-894e-9ab7d091e313\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " Oct 31 01:57:03.856595 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.856523 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t\" (UniqueName: \"kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t\") pod \"b7a85254-6ae0-4990-894e-9ab7d091e313\" (UID: \"b7a85254-6ae0-4990-894e-9ab7d091e313\") " Oct 31 01:57:03.856779 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.856753 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca" pod "b7a85254-6ae0-4990-894e-9ab7d091e313" (UID: "b7a85254-6ae0-4990-894e-9ab7d091e313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:57:03.857027 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.857004 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:03.858755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.858729 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca" pod "b7a85254-6ae0-4990-894e-9ab7d091e313" (UID: "b7a85254-6ae0-4990-894e-9ab7d091e313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:57:03.858755 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.858734 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t" pod "b7a85254-6ae0-4990-894e-9ab7d091e313" (UID: "b7a85254-6ae0-4990-894e-9ab7d091e313"). InnerVolumeSpecName "kube-api-access-mtz4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:57:03.958582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.958502 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7a85254-6ae0-4990-894e-9ab7d091e313-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:03.958582 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:03.958533 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz4t\" (UniqueName: \"kubernetes.io/projected/b7a85254-6ae0-4990-894e-9ab7d091e313-kube-api-access-mtz4t\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:04.659600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:04.659507 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" event={"ID":"b7a85254-6ae0-4990-894e-9ab7d091e313","Type":"ContainerDied","Data":"0be43bf1ab00b7c2f0595036d57ad607dcf8c704d829750f8dd410cb08df4c30"} Oct 31 01:57:04.659600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:04.659581 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0be43bf1ab00b7c2f0595036d57ad607dcf8c704d829750f8dd410cb08df4c30" Oct 31 01:57:04.659600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:04.659517 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx" Oct 31 01:57:11.425232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.425198 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9"] Oct 31 01:57:11.425783 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.425768 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b7a85254-6ae0-4990-894e-9ab7d091e313" containerName="revoke-keys" Oct 31 01:57:11.425823 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.425785 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a85254-6ae0-4990-894e-9ab7d091e313" containerName="revoke-keys" Oct 31 01:57:11.425930 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.425919 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a85254-6ae0-4990-894e-9ab7d091e313" containerName="revoke-keys" Oct 31 01:57:11.430497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.430474 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.531963 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.531930 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:57:11.532142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.531990 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:57:11.534237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.534218 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9"] Oct 31 01:57:11.543565 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.543542 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwtk\" (UniqueName: \"kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.543697 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.543681 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.543766 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.543750 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.645258 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.645228 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.645438 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.645337 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwtk\" (UniqueName: \"kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.645480 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.645462 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.646136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.646113 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.647849 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.647826 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.701376 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.701307 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwtk\" (UniqueName: \"kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk\") pod \"maas-api-revoke-sub-b9957577-sxgp9\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:11.744715 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:11.744689 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:12.121284 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:12.121255 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9"] Oct 31 01:57:12.122581 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:57:12.122555 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb59daab_b613_4656_a88d_1fa01d43e19f.slice/crio-1f197df5dea83f61e8da2bf0283b166d57b389478f12b762255c3d0b19f528c1 WatchSource:0}: Error finding container 1f197df5dea83f61e8da2bf0283b166d57b389478f12b762255c3d0b19f528c1: Status 404 returned error can't find the container with id 1f197df5dea83f61e8da2bf0283b166d57b389478f12b762255c3d0b19f528c1 Oct 31 01:57:12.707342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:12.707246 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" event={"ID":"bb59daab-b613-4656-a88d-1fa01d43e19f","Type":"ContainerStarted","Data":"9ceca2d895a3d50b9d1cc0d1a2614951a88786f8c81ffbc8b13d973841576daf"} Oct 31 01:57:12.707342 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:12.707311 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" event={"ID":"bb59daab-b613-4656-a88d-1fa01d43e19f","Type":"ContainerStarted","Data":"1f197df5dea83f61e8da2bf0283b166d57b389478f12b762255c3d0b19f528c1"} Oct 31 01:57:12.820797 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:12.820738 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" podStartSLOduration=1.820722704 podStartE2EDuration="1.820722704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:12.818606751 +0000 UTC m=+3604.625574295" watchObservedRunningTime="2026-10-31 01:57:12.820722704 +0000 UTC m=+3604.627690288" Oct 31 01:57:13.718272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:13.718237 5756 generic.go:356] "Generic (PLEG): container finished" podID="bb59daab-b613-4656-a88d-1fa01d43e19f" containerID="9ceca2d895a3d50b9d1cc0d1a2614951a88786f8c81ffbc8b13d973841576daf" exitCode=0 Oct 31 01:57:13.718782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:13.718383 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" event={"ID":"bb59daab-b613-4656-a88d-1fa01d43e19f","Type":"ContainerDied","Data":"9ceca2d895a3d50b9d1cc0d1a2614951a88786f8c81ffbc8b13d973841576daf"} Oct 31 01:57:14.556856 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.556819 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:57:14.884152 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.884128 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:14.988486 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.988455 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca\") pod \"bb59daab-b613-4656-a88d-1fa01d43e19f\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " Oct 31 01:57:14.988703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.988535 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca\") pod \"bb59daab-b613-4656-a88d-1fa01d43e19f\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " Oct 31 01:57:14.988703 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.988661 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk\" (UniqueName: \"kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk\") pod \"bb59daab-b613-4656-a88d-1fa01d43e19f\" (UID: \"bb59daab-b613-4656-a88d-1fa01d43e19f\") " Oct 31 01:57:14.988971 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.988936 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca" pod "bb59daab-b613-4656-a88d-1fa01d43e19f" (UID: "bb59daab-b613-4656-a88d-1fa01d43e19f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:57:14.989237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.989211 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:14.991025 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.990997 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk" pod "bb59daab-b613-4656-a88d-1fa01d43e19f" (UID: "bb59daab-b613-4656-a88d-1fa01d43e19f"). InnerVolumeSpecName "kube-api-access-gtwtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:57:14.991121 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:14.991023 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca" pod "bb59daab-b613-4656-a88d-1fa01d43e19f" (UID: "bb59daab-b613-4656-a88d-1fa01d43e19f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:57:15.090925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:15.090890 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb59daab-b613-4656-a88d-1fa01d43e19f-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:15.090925 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:15.090920 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwtk\" (UniqueName: \"kubernetes.io/projected/bb59daab-b613-4656-a88d-1fa01d43e19f-kube-api-access-gtwtk\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:15.734190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:15.734095 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" event={"ID":"bb59daab-b613-4656-a88d-1fa01d43e19f","Type":"ContainerDied","Data":"1f197df5dea83f61e8da2bf0283b166d57b389478f12b762255c3d0b19f528c1"} Oct 31 01:57:15.734190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:15.734138 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9" Oct 31 01:57:15.734190 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:15.734154 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f197df5dea83f61e8da2bf0283b166d57b389478f12b762255c3d0b19f528c1" Oct 31 01:57:32.067307 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:57:32.067271 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:57:42.069180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:42.069143 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr"] Oct 31 01:57:57.090005 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.089965 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" podUID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" containerName="main" containerID="cri-o://2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52" gracePeriod=45 Oct 31 01:57:57.223981 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.223940 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" podUID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.212:8000/ready\": dial tcp 10.134.0.212:8000: connect: connection refused" Oct 31 01:57:57.362439 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.362377 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:57:57.379644 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.379617 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm\") pod \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " Oct 31 01:57:57.379787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.379677 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf\" (UniqueName: \"kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf\") pod \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " Oct 31 01:57:57.379787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.379728 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs\" (UniqueName: \"kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs\") pod \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " Oct 31 01:57:57.379787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.379753 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home\") pod \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " Oct 31 01:57:57.379787 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.379781 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache\") pod \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " Oct 31 01:57:57.380036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.379806 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir\") pod \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\" (UID: \"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a\") " Oct 31 01:57:57.380140 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.380115 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home" pod "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" (UID: "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:57.380291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.380137 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache" pod "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" (UID: "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:57.380291 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.380207 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir" pod "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" (UID: "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:57.381915 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.381895 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm" pod "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" (UID: "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:57.382519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.382497 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs" pod "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" (UID: "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:57:57.382618 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.382540 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf" pod "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" (UID: "05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a"). InnerVolumeSpecName "kube-api-access-gpjnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:57:57.481552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.481519 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:57.481552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.481550 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpjnf\" (UniqueName: \"kubernetes.io/projected/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-kube-api-access-gpjnf\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:57.481552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.481560 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:57.481774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.481569 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:57.481774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.481577 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:57.481774 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.481590 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:57:57.988331 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.988299 5756 generic.go:356] "Generic (PLEG): container finished" podID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" containerID="2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52" exitCode=0 Oct 31 01:57:57.988544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.988355 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" Oct 31 01:57:57.988544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.988385 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" event={"ID":"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a","Type":"ContainerDied","Data":"2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52"} Oct 31 01:57:57.988544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.988432 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr" event={"ID":"05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a","Type":"ContainerDied","Data":"44e07770b34650bb9c002f047bdb27c9432d881c3ef9d8415bd69b65ac66d571"} Oct 31 01:57:57.988544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.988456 5756 scope.go:122] "RemoveContainer" containerID="2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52" Oct 31 01:57:57.998912 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.998896 5756 scope.go:122] "RemoveContainer" containerID="2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52" Oct 31 01:57:57.999155 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:57:57.999137 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52\": container with ID starting with 2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52 not found: ID does not exist" containerID="2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52" Oct 31 01:57:57.999219 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:57.999168 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52"} err="failed to get container status \"2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52\": rpc error: code = NotFound desc = could not find container \"2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52\": container with ID starting with 2215163d3b3136467ea45bbc5fdf7f3dbb55f7290d4dae1f102be63c026f1a52 not found: ID does not exist" Oct 31 01:57:58.166018 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:58.165986 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr"] Oct 31 01:57:58.214945 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:58.214915 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-083e17/auth-test-model-083e17-kserve-6c56744bfc-jmxdr"] Oct 31 01:57:58.312844 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:57:58.312813 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" path="/var/lib/kubelet/pods/05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a/volumes" Oct 31 01:58:02.067000 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:58:02.066962 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:58:11.644828 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:11.644793 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:58:18.647531 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.647491 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx"] Oct 31 01:58:18.648075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.648031 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bb59daab-b613-4656-a88d-1fa01d43e19f" containerName="revoke-keys" Oct 31 01:58:18.648075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.648044 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb59daab-b613-4656-a88d-1fa01d43e19f" containerName="revoke-keys" Oct 31 01:58:18.648075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.648064 5756 cpu_manager.go:414] "containerMap: removing container" podUID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" containerName="main" Oct 31 01:58:18.648075 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.648069 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" containerName="main" Oct 31 01:58:18.648232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.648156 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a3a1e2-dfd8-4d4d-aa27-7cb9b502324a" containerName="main" Oct 31 01:58:18.648232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.648171 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb59daab-b613-4656-a88d-1fa01d43e19f" containerName="revoke-keys" Oct 31 01:58:18.652736 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.652707 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.706446 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.706414 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx"] Oct 31 01:58:18.753015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.752990 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:58:18.753015 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.753003 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:58:18.831373 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.831339 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmklq\" (UniqueName: \"kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.831564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.831456 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.831564 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.831510 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.932427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.932353 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.932427 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.932407 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.932629 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.932505 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmklq\" (UniqueName: \"kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.933200 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.933181 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.934888 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.934851 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:18.988431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:18.988397 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmklq\" (UniqueName: \"kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq\") pod \"maas-api-revoke-sub-e7968cb5-2nbtx\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:19.265836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:19.265761 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:19.638710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:19.638680 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx"] Oct 31 01:58:19.640244 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:58:19.640216 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d1073b_a0d1_4c04_b313_39495ea53b45.slice/crio-638148acce1261b82c3f71de093c820ebdc72f95ef8e2759cb93339dee332bc5 WatchSource:0}: Error finding container 638148acce1261b82c3f71de093c820ebdc72f95ef8e2759cb93339dee332bc5: Status 404 returned error can't find the container with id 638148acce1261b82c3f71de093c820ebdc72f95ef8e2759cb93339dee332bc5 Oct 31 01:58:20.127812 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.127773 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" event={"ID":"c0d1073b-a0d1-4c04-b313-39495ea53b45","Type":"ContainerStarted","Data":"37eb6f98871005d0c78c09d78efc8250da73bf238e66db38500663bd5853098b"} Oct 31 01:58:20.128254 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.127839 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" event={"ID":"c0d1073b-a0d1-4c04-b313-39495ea53b45","Type":"ContainerStarted","Data":"638148acce1261b82c3f71de093c820ebdc72f95ef8e2759cb93339dee332bc5"} Oct 31 01:58:20.241308 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.241209 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" podStartSLOduration=2.241194024 podStartE2EDuration="2.241194024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:20.24038502 +0000 UTC m=+3672.047352569" watchObservedRunningTime="2026-10-31 01:58:20.241194024 +0000 UTC m=+3672.048161566" Oct 31 01:58:20.264229 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.264185 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv"] Oct 31 01:58:20.268216 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.268198 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.353552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.353519 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv"] Oct 31 01:58:20.453763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.453713 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.453946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.453784 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkqz\" (UniqueName: \"kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.453946 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.453935 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.555098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.555068 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkqz\" (UniqueName: \"kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.555409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.555148 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.555409 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.555249 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.555767 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.555739 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.557692 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.557667 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.612153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.612125 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkqz\" (UniqueName: \"kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz\") pod \"maas-api-revoke-sub-584140cf-phjnv\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:20.883763 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:20.883678 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:21.136273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:21.136075 5756 generic.go:356] "Generic (PLEG): container finished" podID="c0d1073b-a0d1-4c04-b313-39495ea53b45" containerID="37eb6f98871005d0c78c09d78efc8250da73bf238e66db38500663bd5853098b" exitCode=0 Oct 31 01:58:21.136273 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:21.136142 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" event={"ID":"c0d1073b-a0d1-4c04-b313-39495ea53b45","Type":"ContainerDied","Data":"37eb6f98871005d0c78c09d78efc8250da73bf238e66db38500663bd5853098b"} Oct 31 01:58:21.261388 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:58:21.261345 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1b753f_180a_4e02_bc5b_aaad131b7dda.slice/crio-16572e98effe33b212dd32a06eb3f9dafebbcc9896a158a18f49e6b9c4330a03 WatchSource:0}: Error finding container 16572e98effe33b212dd32a06eb3f9dafebbcc9896a158a18f49e6b9c4330a03: Status 404 returned error can't find the container with id 16572e98effe33b212dd32a06eb3f9dafebbcc9896a158a18f49e6b9c4330a03 Oct 31 01:58:21.299606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:21.299574 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv"] Oct 31 01:58:22.143727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.143686 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" event={"ID":"9f1b753f-180a-4e02-bc5b-aaad131b7dda","Type":"ContainerStarted","Data":"e62d765ecf2ad3ae1314c7a0838fa57966b018b6416d11e3b45f9059934a9a0a"} Oct 31 01:58:22.144232 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.143784 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" event={"ID":"9f1b753f-180a-4e02-bc5b-aaad131b7dda","Type":"ContainerStarted","Data":"16572e98effe33b212dd32a06eb3f9dafebbcc9896a158a18f49e6b9c4330a03"} Oct 31 01:58:22.256318 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.256264 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" podStartSLOduration=2.256249407 podStartE2EDuration="2.256249407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:22.255447413 +0000 UTC m=+3674.062414957" watchObservedRunningTime="2026-10-31 01:58:22.256249407 +0000 UTC m=+3674.063216973" Oct 31 01:58:22.301646 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.301623 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:22.477228 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.477150 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq\" (UniqueName: \"kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq\") pod \"c0d1073b-a0d1-4c04-b313-39495ea53b45\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " Oct 31 01:58:22.477405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.477235 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca\") pod \"c0d1073b-a0d1-4c04-b313-39495ea53b45\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " Oct 31 01:58:22.477405 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.477332 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca\") pod \"c0d1073b-a0d1-4c04-b313-39495ea53b45\" (UID: \"c0d1073b-a0d1-4c04-b313-39495ea53b45\") " Oct 31 01:58:22.477694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.477669 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca" pod "c0d1073b-a0d1-4c04-b313-39495ea53b45" (UID: "c0d1073b-a0d1-4c04-b313-39495ea53b45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:58:22.479351 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.479321 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq" pod "c0d1073b-a0d1-4c04-b313-39495ea53b45" (UID: "c0d1073b-a0d1-4c04-b313-39495ea53b45"). InnerVolumeSpecName "kube-api-access-fmklq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:22.479457 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.479386 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca" pod "c0d1073b-a0d1-4c04-b313-39495ea53b45" (UID: "c0d1073b-a0d1-4c04-b313-39495ea53b45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:58:22.578355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.578305 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmklq\" (UniqueName: \"kubernetes.io/projected/c0d1073b-a0d1-4c04-b313-39495ea53b45-kube-api-access-fmklq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:22.578355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.578351 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:22.578355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:22.578365 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d1073b-a0d1-4c04-b313-39495ea53b45-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:23.151418 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:23.151389 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" Oct 31 01:58:23.151903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:23.151414 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx" event={"ID":"c0d1073b-a0d1-4c04-b313-39495ea53b45","Type":"ContainerDied","Data":"638148acce1261b82c3f71de093c820ebdc72f95ef8e2759cb93339dee332bc5"} Oct 31 01:58:23.151903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:23.151471 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638148acce1261b82c3f71de093c820ebdc72f95ef8e2759cb93339dee332bc5" Oct 31 01:58:23.153279 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:23.153250 5756 generic.go:356] "Generic (PLEG): container finished" podID="9f1b753f-180a-4e02-bc5b-aaad131b7dda" containerID="e62d765ecf2ad3ae1314c7a0838fa57966b018b6416d11e3b45f9059934a9a0a" exitCode=0 Oct 31 01:58:23.153415 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:23.153292 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" event={"ID":"9f1b753f-180a-4e02-bc5b-aaad131b7dda","Type":"ContainerDied","Data":"e62d765ecf2ad3ae1314c7a0838fa57966b018b6416d11e3b45f9059934a9a0a"} Oct 31 01:58:24.304355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.304333 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:24.395424 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.395383 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca\") pod \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " Oct 31 01:58:24.395725 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.395703 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca" pod "9f1b753f-180a-4e02-bc5b-aaad131b7dda" (UID: "9f1b753f-180a-4e02-bc5b-aaad131b7dda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:58:24.496924 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.496803 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz\" (UniqueName: \"kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz\") pod \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " Oct 31 01:58:24.497113 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.496971 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca\") pod \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\" (UID: \"9f1b753f-180a-4e02-bc5b-aaad131b7dda\") " Oct 31 01:58:24.497353 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.497334 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:24.499098 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.499069 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca" pod "9f1b753f-180a-4e02-bc5b-aaad131b7dda" (UID: "9f1b753f-180a-4e02-bc5b-aaad131b7dda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:58:24.499197 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.499125 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz" pod "9f1b753f-180a-4e02-bc5b-aaad131b7dda" (UID: "9f1b753f-180a-4e02-bc5b-aaad131b7dda"). InnerVolumeSpecName "kube-api-access-mxkqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:24.598680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.598644 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxkqz\" (UniqueName: \"kubernetes.io/projected/9f1b753f-180a-4e02-bc5b-aaad131b7dda-kube-api-access-mxkqz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:24.598680 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:24.598672 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1b753f-180a-4e02-bc5b-aaad131b7dda-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:25.167148 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:25.167117 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" Oct 31 01:58:25.167324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:25.167155 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv" event={"ID":"9f1b753f-180a-4e02-bc5b-aaad131b7dda","Type":"ContainerDied","Data":"16572e98effe33b212dd32a06eb3f9dafebbcc9896a158a18f49e6b9c4330a03"} Oct 31 01:58:25.167324 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:25.167204 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16572e98effe33b212dd32a06eb3f9dafebbcc9896a158a18f49e6b9c4330a03" Oct 31 01:58:29.388886 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.388832 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m"] Oct 31 01:58:29.389355 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.389341 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9f1b753f-180a-4e02-bc5b-aaad131b7dda" containerName="revoke-keys" Oct 31 01:58:29.389402 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.389357 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1b753f-180a-4e02-bc5b-aaad131b7dda" containerName="revoke-keys" Oct 31 01:58:29.389402 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.389392 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c0d1073b-a0d1-4c04-b313-39495ea53b45" containerName="revoke-keys" Oct 31 01:58:29.389402 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.389397 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d1073b-a0d1-4c04-b313-39495ea53b45" containerName="revoke-keys" Oct 31 01:58:29.389502 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.389496 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1b753f-180a-4e02-bc5b-aaad131b7dda" containerName="revoke-keys" Oct 31 01:58:29.389539 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.389511 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d1073b-a0d1-4c04-b313-39495ea53b45" containerName="revoke-keys" Oct 31 01:58:29.393118 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.393101 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.448952 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.448922 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m"] Oct 31 01:58:29.455175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.455143 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlr69\" (UniqueName: \"kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.455339 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.455202 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.455533 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.455510 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.494550 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.494517 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 01:58:29.494709 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.494602 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 01:58:29.557237 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.557201 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.557430 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.557266 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlr69\" (UniqueName: \"kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.557430 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.557299 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.558000 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.557981 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.559756 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.559733 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.613328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.613301 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlr69\" (UniqueName: \"kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69\") pod \"maas-api-revoke-sub-581f4412-b8p9m\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:29.706548 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:29.706469 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:30.082136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:30.082105 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m"] Oct 31 01:58:30.083668 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:58:30.083643 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0550d329_d52c_4f04_96bd_a3956205046b.slice/crio-1903fdc35042de155f43797deec64fa371c56dfeead87afffe49484c9f520d89 WatchSource:0}: Error finding container 1903fdc35042de155f43797deec64fa371c56dfeead87afffe49484c9f520d89: Status 404 returned error can't find the container with id 1903fdc35042de155f43797deec64fa371c56dfeead87afffe49484c9f520d89 Oct 31 01:58:30.198428 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:30.198395 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" event={"ID":"0550d329-d52c-4f04-96bd-a3956205046b","Type":"ContainerStarted","Data":"1903fdc35042de155f43797deec64fa371c56dfeead87afffe49484c9f520d89"} Oct 31 01:58:31.007723 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.007693 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds"] Oct 31 01:58:31.011636 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.011621 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.067598 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.067560 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds"] Oct 31 01:58:31.080699 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.080669 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.080841 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.080779 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.080958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.080942 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np28z\" (UniqueName: \"kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.182172 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.182089 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.182328 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.182234 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np28z\" (UniqueName: \"kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.182377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.182331 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.182798 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.182775 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.184752 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.184731 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.215683 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.215651 5756 generic.go:356] "Generic (PLEG): container finished" podID="0550d329-d52c-4f04-96bd-a3956205046b" containerID="176451f867fa22bd4689545a6a93a8ecadb8a6422a3737ed94336e7d29208a8d" exitCode=0 Oct 31 01:58:31.215884 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.215736 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" event={"ID":"0550d329-d52c-4f04-96bd-a3956205046b","Type":"ContainerDied","Data":"176451f867fa22bd4689545a6a93a8ecadb8a6422a3737ed94336e7d29208a8d"} Oct 31 01:58:31.238587 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.238560 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np28z\" (UniqueName: \"kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z\") pod \"maas-api-revoke-sub-da5b671f-7p2ds\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.324820 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.324790 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:31.702489 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:31.702467 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds"] Oct 31 01:58:31.704604 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:58:31.704578 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4b519e_d61f_45ba_bbba_635c9579b891.slice/crio-40fcd63ef6b181d7773bc3580202dd7ce5dfb46eb94de5995f8f42f27b8a9816 WatchSource:0}: Error finding container 40fcd63ef6b181d7773bc3580202dd7ce5dfb46eb94de5995f8f42f27b8a9816: Status 404 returned error can't find the container with id 40fcd63ef6b181d7773bc3580202dd7ce5dfb46eb94de5995f8f42f27b8a9816 Oct 31 01:58:32.067972 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:58:32.067934 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:58:32.223247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.223211 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" event={"ID":"3f4b519e-d61f-45ba-bbba-635c9579b891","Type":"ContainerStarted","Data":"bd5dcf3f77f6a49ecf7480101b9659ec049a5acd9c7fd9456a7e78ab0c042653"} Oct 31 01:58:32.223400 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.223266 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" event={"ID":"3f4b519e-d61f-45ba-bbba-635c9579b891","Type":"ContainerStarted","Data":"40fcd63ef6b181d7773bc3580202dd7ce5dfb46eb94de5995f8f42f27b8a9816"} Oct 31 01:58:32.336916 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.336809 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" podStartSLOduration=2.336794104 podStartE2EDuration="2.336794104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:32.334800983 +0000 UTC m=+3684.141768526" watchObservedRunningTime="2026-10-31 01:58:32.336794104 +0000 UTC m=+3684.143761647" Oct 31 01:58:32.383263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.383231 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:32.498494 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.498464 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca\") pod \"0550d329-d52c-4f04-96bd-a3956205046b\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " Oct 31 01:58:32.498656 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.498512 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca\") pod \"0550d329-d52c-4f04-96bd-a3956205046b\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " Oct 31 01:58:32.498734 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.498719 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69\" (UniqueName: \"kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69\") pod \"0550d329-d52c-4f04-96bd-a3956205046b\" (UID: \"0550d329-d52c-4f04-96bd-a3956205046b\") " Oct 31 01:58:32.500012 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.499969 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca" pod "0550d329-d52c-4f04-96bd-a3956205046b" (UID: "0550d329-d52c-4f04-96bd-a3956205046b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:58:32.501704 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.501676 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69" pod "0550d329-d52c-4f04-96bd-a3956205046b" (UID: "0550d329-d52c-4f04-96bd-a3956205046b"). InnerVolumeSpecName "kube-api-access-xlr69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:32.501801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.501735 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca" pod "0550d329-d52c-4f04-96bd-a3956205046b" (UID: "0550d329-d52c-4f04-96bd-a3956205046b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:58:32.600687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.600610 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:32.600687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.600642 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0550d329-d52c-4f04-96bd-a3956205046b-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:32.600687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:32.600652 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlr69\" (UniqueName: \"kubernetes.io/projected/0550d329-d52c-4f04-96bd-a3956205046b-kube-api-access-xlr69\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:33.230497 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:33.230460 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" Oct 31 01:58:33.230948 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:33.230497 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m" event={"ID":"0550d329-d52c-4f04-96bd-a3956205046b","Type":"ContainerDied","Data":"1903fdc35042de155f43797deec64fa371c56dfeead87afffe49484c9f520d89"} Oct 31 01:58:33.230948 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:33.230556 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1903fdc35042de155f43797deec64fa371c56dfeead87afffe49484c9f520d89" Oct 31 01:58:33.232126 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:33.232102 5756 generic.go:356] "Generic (PLEG): container finished" podID="3f4b519e-d61f-45ba-bbba-635c9579b891" containerID="bd5dcf3f77f6a49ecf7480101b9659ec049a5acd9c7fd9456a7e78ab0c042653" exitCode=0 Oct 31 01:58:33.232275 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:33.232192 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" event={"ID":"3f4b519e-d61f-45ba-bbba-635c9579b891","Type":"ContainerDied","Data":"bd5dcf3f77f6a49ecf7480101b9659ec049a5acd9c7fd9456a7e78ab0c042653"} Oct 31 01:58:34.391981 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.391947 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:34.522892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.522807 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca\") pod \"3f4b519e-d61f-45ba-bbba-635c9579b891\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " Oct 31 01:58:34.522892 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.522856 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z\" (UniqueName: \"kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z\") pod \"3f4b519e-d61f-45ba-bbba-635c9579b891\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " Oct 31 01:58:34.523089 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.522941 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca\") pod \"3f4b519e-d61f-45ba-bbba-635c9579b891\" (UID: \"3f4b519e-d61f-45ba-bbba-635c9579b891\") " Oct 31 01:58:34.523247 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.523222 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca" pod "3f4b519e-d61f-45ba-bbba-635c9579b891" (UID: "3f4b519e-d61f-45ba-bbba-635c9579b891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:58:34.523516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.523497 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:34.525276 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.525251 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z" pod "3f4b519e-d61f-45ba-bbba-635c9579b891" (UID: "3f4b519e-d61f-45ba-bbba-635c9579b891"). InnerVolumeSpecName "kube-api-access-np28z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:34.525484 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.525465 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca" pod "3f4b519e-d61f-45ba-bbba-635c9579b891" (UID: "3f4b519e-d61f-45ba-bbba-635c9579b891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:58:34.624693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.624658 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np28z\" (UniqueName: \"kubernetes.io/projected/3f4b519e-d61f-45ba-bbba-635c9579b891-kube-api-access-np28z\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:34.624693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:34.624686 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4b519e-d61f-45ba-bbba-635c9579b891-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:58:35.246414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:35.246370 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" event={"ID":"3f4b519e-d61f-45ba-bbba-635c9579b891","Type":"ContainerDied","Data":"40fcd63ef6b181d7773bc3580202dd7ce5dfb46eb94de5995f8f42f27b8a9816"} Oct 31 01:58:35.246621 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:35.246422 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds" Oct 31 01:58:35.246621 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:35.246438 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40fcd63ef6b181d7773bc3580202dd7ce5dfb46eb94de5995f8f42f27b8a9816" Oct 31 01:58:50.436239 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:50.436162 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r"] Oct 31 01:58:52.269883 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:52.269839 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd"] Oct 31 01:58:53.762519 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:53.762488 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 01:58:59.216777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.216737 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw"] Oct 31 01:58:59.217394 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.217373 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3f4b519e-d61f-45ba-bbba-635c9579b891" containerName="revoke-keys" Oct 31 01:58:59.217552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.217396 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4b519e-d61f-45ba-bbba-635c9579b891" containerName="revoke-keys" Oct 31 01:58:59.217552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.217418 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0550d329-d52c-4f04-96bd-a3956205046b" containerName="revoke-keys" Oct 31 01:58:59.217552 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.217424 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0550d329-d52c-4f04-96bd-a3956205046b" containerName="revoke-keys" Oct 31 01:58:59.217695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.217621 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4b519e-d61f-45ba-bbba-635c9579b891" containerName="revoke-keys" Oct 31 01:58:59.217695 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.217638 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0550d329-d52c-4f04-96bd-a3956205046b" containerName="revoke-keys" Oct 31 01:58:59.221757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.221740 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.301735 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.301692 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6caf19-openshift-default-dockercfg-22nqd\"" Oct 31 01:58:59.333226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333190 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333229 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333274 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333306 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333375 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333393 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333414 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333449 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.333479 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.333465 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl884\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.336359 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.336320 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw"] Oct 31 01:58:59.434672 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.434639 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.434887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.434684 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.434887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.434825 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.434902 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.434939 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.434989 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435016 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl884\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435096 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435189 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435228 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435233 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435498 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435275 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435544 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435503 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.435666 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.435640 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.437551 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.437531 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.437777 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.437758 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.495019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.494946 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.495185 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.495159 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl884\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884\") pod \"e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.539370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.539334 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:58:59.979039 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.979012 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw"] Oct 31 01:58:59.981572 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:58:59.981547 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e05eef2_76d3_4dec_bea6_bef86f3fdf31.slice/crio-2b247285831c0a485f090774604b44c54e7c37ef564f46e27fc214dea82c6f13 WatchSource:0}: Error finding container 2b247285831c0a485f090774604b44c54e7c37ef564f46e27fc214dea82c6f13: Status 404 returned error can't find the container with id 2b247285831c0a485f090774604b44c54e7c37ef564f46e27fc214dea82c6f13 Oct 31 01:58:59.984313 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.984282 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:58:59.984382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.984350 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:58:59.984429 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:58:59.984379 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 01:59:00.412171 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:00.412132 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" event={"ID":"3e05eef2-76d3-4dec-bea6-bef86f3fdf31","Type":"ContainerStarted","Data":"285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90"} Oct 31 01:59:00.412609 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:00.412187 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" event={"ID":"3e05eef2-76d3-4dec-bea6-bef86f3fdf31","Type":"ContainerStarted","Data":"2b247285831c0a485f090774604b44c54e7c37ef564f46e27fc214dea82c6f13"} Oct 31 01:59:00.530836 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:00.530778 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podStartSLOduration=1.530761816 podStartE2EDuration="1.530761816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:00.527802716 +0000 UTC m=+3712.334770295" watchObservedRunningTime="2026-10-31 01:59:00.530761816 +0000 UTC m=+3712.337729358" Oct 31 01:59:00.539887 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:00.539845 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:59:01.545515 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:01.545479 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:59:02.068029 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:59:02.067989 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:59:02.423881 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:02.423791 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:59:02.424847 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:02.424831 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 01:59:03.231033 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.230993 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr"] Oct 31 01:59:03.235175 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.235154 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.283462 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.283430 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.283669 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.283486 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnkr\" (UniqueName: \"kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr\") pod \"payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.288226 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.288201 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6c6990\"" Oct 31 01:59:03.288487 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.288464 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6c6990-dockercfg-7kndg\"" Oct 31 01:59:03.291851 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.291827 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr"] Oct 31 01:59:03.374838 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.374805 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw"] Oct 31 01:59:03.379108 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.379087 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.384431 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.384401 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.384557 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.384453 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnkr\" (UniqueName: \"kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr\") pod \"payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.385142 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.385123 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.435727 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.435693 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw"] Oct 31 01:59:03.442180 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.442155 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnkr\" (UniqueName: \"kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr\") pod \"payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.485493 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.485399 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqbb\" (UniqueName: \"kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb\") pod \"payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.485710 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.485678 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.549316 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.549279 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:03.587383 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.587352 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqbb\" (UniqueName: \"kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb\") pod \"payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.587542 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.587453 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.588207 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.588164 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.644193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.644162 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqbb\" (UniqueName: \"kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb\") pod \"payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.693782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.693752 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:03.926965 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:03.926936 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr"] Oct 31 01:59:03.928856 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:59:03.928830 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006578cd_b9d3_405e_a323_4ae79d19a5ee.slice/crio-050e9e851a8bc99ee8f079b6d6d5bfa5bbcac429dc1f70b89d810ae16ae1cb3d WatchSource:0}: Error finding container 050e9e851a8bc99ee8f079b6d6d5bfa5bbcac429dc1f70b89d810ae16ae1cb3d: Status 404 returned error can't find the container with id 050e9e851a8bc99ee8f079b6d6d5bfa5bbcac429dc1f70b89d810ae16ae1cb3d Oct 31 01:59:04.072694 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.072669 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw"] Oct 31 01:59:04.074686 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:59:04.074659 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa99ba2_4df3_41b0_92d0_92d445980db2.slice/crio-5a3d5911a132fca0440f7a481e94ed2de866081ad076d3a879822d6b2ab5beb4 WatchSource:0}: Error finding container 5a3d5911a132fca0440f7a481e94ed2de866081ad076d3a879822d6b2ab5beb4: Status 404 returned error can't find the container with id 5a3d5911a132fca0440f7a481e94ed2de866081ad076d3a879822d6b2ab5beb4 Oct 31 01:59:04.438154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.438079 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" event={"ID":"006578cd-b9d3-405e-a323-4ae79d19a5ee","Type":"ContainerStarted","Data":"84a8fa74aa8356fd78e53fb22655bbbccf8f4402703e638b4faea92d1bf8c820"} Oct 31 01:59:04.438154 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.438136 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" event={"ID":"006578cd-b9d3-405e-a323-4ae79d19a5ee","Type":"ContainerStarted","Data":"050e9e851a8bc99ee8f079b6d6d5bfa5bbcac429dc1f70b89d810ae16ae1cb3d"} Oct 31 01:59:04.438655 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.438265 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:04.439913 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.439886 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" event={"ID":"6aa99ba2-4df3-41b0-92d0-92d445980db2","Type":"ContainerStarted","Data":"e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86"} Oct 31 01:59:04.440045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.439934 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" event={"ID":"6aa99ba2-4df3-41b0-92d0-92d445980db2","Type":"ContainerStarted","Data":"5a3d5911a132fca0440f7a481e94ed2de866081ad076d3a879822d6b2ab5beb4"} Oct 31 01:59:04.440110 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.440085 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:04.553903 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.553830 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" podStartSLOduration=1.553815235 podStartE2EDuration="1.553815235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:04.552029991 +0000 UTC m=+3716.358997538" watchObservedRunningTime="2026-10-31 01:59:04.553815235 +0000 UTC m=+3716.360782778" Oct 31 01:59:04.664600 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:04.664539 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" podStartSLOduration=1.66452432 podStartE2EDuration="1.66452432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:04.663464669 +0000 UTC m=+3716.470432212" watchObservedRunningTime="2026-10-31 01:59:04.66452432 +0000 UTC m=+3716.471491862" Oct 31 01:59:05.457196 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.457153 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" podUID="9b89314f-373c-47c6-b499-0abfcb749301" containerName="main" containerID="cri-o://743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a" gracePeriod=45 Oct 31 01:59:05.721364 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.721302 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:59:05.811606 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.811576 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs\") pod \"9b89314f-373c-47c6-b499-0abfcb749301\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " Oct 31 01:59:05.811757 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.811695 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache\") pod \"9b89314f-373c-47c6-b499-0abfcb749301\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " Oct 31 01:59:05.811825 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.811787 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home\") pod \"9b89314f-373c-47c6-b499-0abfcb749301\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " Oct 31 01:59:05.811825 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.811815 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir\") pod \"9b89314f-373c-47c6-b499-0abfcb749301\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " Oct 31 01:59:05.811944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.811836 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm\") pod \"9b89314f-373c-47c6-b499-0abfcb749301\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " Oct 31 01:59:05.811944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.811887 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v\" (UniqueName: \"kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v\") pod \"9b89314f-373c-47c6-b499-0abfcb749301\" (UID: \"9b89314f-373c-47c6-b499-0abfcb749301\") " Oct 31 01:59:05.812060 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.812033 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache" pod "9b89314f-373c-47c6-b499-0abfcb749301" (UID: "9b89314f-373c-47c6-b499-0abfcb749301"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:05.812211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.812186 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir" pod "9b89314f-373c-47c6-b499-0abfcb749301" (UID: "9b89314f-373c-47c6-b499-0abfcb749301"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:05.812211 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.812199 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home" pod "9b89314f-373c-47c6-b499-0abfcb749301" (UID: "9b89314f-373c-47c6-b499-0abfcb749301"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:05.812693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.812628 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:05.812693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.812654 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:05.812693 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.812670 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:05.814112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.814083 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm" pod "9b89314f-373c-47c6-b499-0abfcb749301" (UID: "9b89314f-373c-47c6-b499-0abfcb749301"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:05.814112 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.814102 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs" pod "9b89314f-373c-47c6-b499-0abfcb749301" (UID: "9b89314f-373c-47c6-b499-0abfcb749301"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:59:05.814241 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.814198 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v" pod "9b89314f-373c-47c6-b499-0abfcb749301" (UID: "9b89314f-373c-47c6-b499-0abfcb749301"). InnerVolumeSpecName "kube-api-access-wn67v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:59:05.913778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.913747 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b89314f-373c-47c6-b499-0abfcb749301-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:05.913778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.913775 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn67v\" (UniqueName: \"kubernetes.io/projected/9b89314f-373c-47c6-b499-0abfcb749301-kube-api-access-wn67v\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:05.913778 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:05.913785 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b89314f-373c-47c6-b499-0abfcb749301-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:06.455341 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.455304 5756 generic.go:356] "Generic (PLEG): container finished" podID="9b89314f-373c-47c6-b499-0abfcb749301" containerID="743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a" exitCode=0 Oct 31 01:59:06.455495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.455363 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" Oct 31 01:59:06.455495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.455392 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" event={"ID":"9b89314f-373c-47c6-b499-0abfcb749301","Type":"ContainerDied","Data":"743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a"} Oct 31 01:59:06.455495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.455439 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r" event={"ID":"9b89314f-373c-47c6-b499-0abfcb749301","Type":"ContainerDied","Data":"abeab07903abc1f59698ab56310637f222fd1dc13b0ca525bd221046e25729cd"} Oct 31 01:59:06.455495 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.455471 5756 scope.go:122] "RemoveContainer" containerID="743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a" Oct 31 01:59:06.465517 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.465340 5756 scope.go:122] "RemoveContainer" containerID="743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a" Oct 31 01:59:06.465758 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:59:06.465608 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a\": container with ID starting with 743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a not found: ID does not exist" containerID="743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a" Oct 31 01:59:06.465758 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.465633 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a"} err="failed to get container status \"743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a\": rpc error: code = NotFound desc = could not find container \"743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a\": container with ID starting with 743b5d18c304d4be85ff7a114012b6c3f44b4bf309fc57adf23e619c19d7db0a not found: ID does not exist" Oct 31 01:59:06.618454 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.618372 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r"] Oct 31 01:59:06.667019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:06.666987 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b992fe/auth-test-model-b992fe-kserve-5bfbbdcdfc-s999r"] Oct 31 01:59:07.293442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.293404 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" podUID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" containerName="main" containerID="cri-o://a7518d8316ba918b70c5c691ede990cbcc33311fbaa5490d087492012bf9fc1b" gracePeriod=45 Oct 31 01:59:07.361615 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.361583 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" podUID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.213:8000/ready\": dial tcp 10.134.0.213:8000: connect: connection refused" Oct 31 01:59:07.467136 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.467101 5756 generic.go:356] "Generic (PLEG): container finished" podID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" containerID="a7518d8316ba918b70c5c691ede990cbcc33311fbaa5490d087492012bf9fc1b" exitCode=0 Oct 31 01:59:07.467491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.467180 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" event={"ID":"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc","Type":"ContainerDied","Data":"a7518d8316ba918b70c5c691ede990cbcc33311fbaa5490d087492012bf9fc1b"} Oct 31 01:59:07.559442 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.559381 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:59:07.632367 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632334 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs\" (UniqueName: \"kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs\") pod \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " Oct 31 01:59:07.632569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632397 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm\") pod \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " Oct 31 01:59:07.632569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632469 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home\") pod \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " Oct 31 01:59:07.632569 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632538 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir\") pod \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " Oct 31 01:59:07.632742 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632612 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts\" (UniqueName: \"kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts\") pod \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " Oct 31 01:59:07.632742 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632712 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache\") pod \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\" (UID: \"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc\") " Oct 31 01:59:07.632855 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632753 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home" pod "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" (UID: "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:07.632855 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.632828 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir" pod "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" (UID: "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:07.633045 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.633019 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache" pod "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" (UID: "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:07.633444 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.633423 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:07.633491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.633453 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:07.633491 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.633467 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:07.634823 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.634799 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm" pod "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" (UID: "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:07.634926 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.634908 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs" pod "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" (UID: "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:59:07.635062 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.635046 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts" pod "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" (UID: "3046b0ec-6da0-4e04-a5f3-e9a729c38fcc"). InnerVolumeSpecName "kube-api-access-mg6ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:59:07.734845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.734817 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:07.734845 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.734844 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6ts\" (UniqueName: \"kubernetes.io/projected/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-kube-api-access-mg6ts\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:07.735067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.734855 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:07.952889 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.952801 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr"] Oct 31 01:59:07.953340 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.953327 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9b89314f-373c-47c6-b499-0abfcb749301" containerName="main" Oct 31 01:59:07.953382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.953342 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b89314f-373c-47c6-b499-0abfcb749301" containerName="main" Oct 31 01:59:07.953382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.953373 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" containerName="main" Oct 31 01:59:07.953382 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.953378 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" containerName="main" Oct 31 01:59:07.953481 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.953474 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b89314f-373c-47c6-b499-0abfcb749301" containerName="main" Oct 31 01:59:07.953516 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.953488 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" containerName="main" Oct 31 01:59:07.956957 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:07.956942 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.009059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.009026 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6caf19\"" Oct 31 01:59:08.009059 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.009026 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6caf19-dockercfg-mfzxc\"" Oct 31 01:59:08.039604 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.039566 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxtkt\" (UniqueName: \"kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt\") pod \"payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.039937 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.039915 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.070040 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.070003 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr"] Oct 31 01:59:08.100840 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.100796 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd"] Oct 31 01:59:08.104827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.104804 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.141546 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.141514 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxtkt\" (UniqueName: \"kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt\") pod \"payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.141754 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.141583 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.142370 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.142349 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.161372 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.161342 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd"] Oct 31 01:59:08.198038 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.198015 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxtkt\" (UniqueName: \"kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt\") pod \"payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.242624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.242548 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-889tc\" (UniqueName: \"kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc\") pod \"payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.242624 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.242598 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.270944 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.270915 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:08.315470 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.315437 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b89314f-373c-47c6-b499-0abfcb749301" path="/var/lib/kubelet/pods/9b89314f-373c-47c6-b499-0abfcb749301/volumes" Oct 31 01:59:08.344036 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.344006 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-889tc\" (UniqueName: \"kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc\") pod \"payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.344186 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.344082 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.345037 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.345014 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.400251 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.400199 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-889tc\" (UniqueName: \"kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc\") pod \"payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.418920 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.418895 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:08.475776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.475692 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" Oct 31 01:59:08.475776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.475691 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd" event={"ID":"3046b0ec-6da0-4e04-a5f3-e9a729c38fcc","Type":"ContainerDied","Data":"4c94632bdc13311c9e4a3a6d020006953d9251c0de0fe9daa669888305f335b1"} Oct 31 01:59:08.475776 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.475761 5756 scope.go:122] "RemoveContainer" containerID="a7518d8316ba918b70c5c691ede990cbcc33311fbaa5490d087492012bf9fc1b" Oct 31 01:59:08.639608 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.639579 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd"] Oct 31 01:59:08.643414 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.643388 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fc058/auth-test-model-7fc058-kserve-57b496f65-kzpcd"] Oct 31 01:59:08.646043 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.646021 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr"] Oct 31 01:59:08.691235 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:59:08.691204 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e28427_6684_4ad9_8978_5e6438db9548.slice/crio-d5995f3071044aa90bfe66faa6ea1d597f561d39db52c2e9d1bc72741503e0f2 WatchSource:0}: Error finding container d5995f3071044aa90bfe66faa6ea1d597f561d39db52c2e9d1bc72741503e0f2: Status 404 returned error can't find the container with id d5995f3071044aa90bfe66faa6ea1d597f561d39db52c2e9d1bc72741503e0f2 Oct 31 01:59:08.811805 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:08.811773 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd"] Oct 31 01:59:08.812930 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:59:08.812907 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac65895_9240_4575_b70f_91ce321af50a.slice/crio-ed0525f543ed3b54a83759abfd7e9dc0ead87e13cdd011a0144c9a0a48c49afb WatchSource:0}: Error finding container ed0525f543ed3b54a83759abfd7e9dc0ead87e13cdd011a0144c9a0a48c49afb: Status 404 returned error can't find the container with id ed0525f543ed3b54a83759abfd7e9dc0ead87e13cdd011a0144c9a0a48c49afb Oct 31 01:59:09.483484 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.483440 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" event={"ID":"6ac65895-9240-4575-b70f-91ce321af50a","Type":"ContainerStarted","Data":"6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93"} Oct 31 01:59:09.484019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.483543 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:09.484019 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.483565 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" event={"ID":"6ac65895-9240-4575-b70f-91ce321af50a","Type":"ContainerStarted","Data":"ed0525f543ed3b54a83759abfd7e9dc0ead87e13cdd011a0144c9a0a48c49afb"} Oct 31 01:59:09.486166 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.486145 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" event={"ID":"98e28427-6684-4ad9-8978-5e6438db9548","Type":"ContainerStarted","Data":"650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205"} Oct 31 01:59:09.486269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.486185 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" event={"ID":"98e28427-6684-4ad9-8978-5e6438db9548","Type":"ContainerStarted","Data":"d5995f3071044aa90bfe66faa6ea1d597f561d39db52c2e9d1bc72741503e0f2"} Oct 31 01:59:09.486269 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.486218 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:09.598381 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.598330 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" podStartSLOduration=1.598316303 podStartE2EDuration="1.598316303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:09.595590912 +0000 UTC m=+3721.402558458" watchObservedRunningTime="2026-10-31 01:59:09.598316303 +0000 UTC m=+3721.405283838" Oct 31 01:59:09.707731 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:09.707638 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" podStartSLOduration=2.707621278 podStartE2EDuration="2.707621278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:09.707201476 +0000 UTC m=+3721.514169020" watchObservedRunningTime="2026-10-31 01:59:09.707621278 +0000 UTC m=+3721.514588824" Oct 31 01:59:10.311344 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:10.311315 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3046b0ec-6da0-4e04-a5f3-e9a729c38fcc" path="/var/lib/kubelet/pods/3046b0ec-6da0-4e04-a5f3-e9a729c38fcc/volumes" Oct 31 01:59:10.585336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:10.585254 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g"] Oct 31 01:59:15.450079 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:15.450047 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 01:59:15.450464 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:15.450222 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 01:59:20.495361 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.495333 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 01:59:20.495827 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.495541 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 01:59:20.601066 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.601022 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997"] Oct 31 01:59:20.606398 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.606379 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.620687 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.618907 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997"] Oct 31 01:59:20.658263 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.658227 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ce8dbb\"" Oct 31 01:59:20.658432 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.658340 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ce8dbb-dockercfg-7cnwc\"" Oct 31 01:59:20.701717 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.701687 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.701906 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.701798 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzq5h\" (UniqueName: \"kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h\") pod \"payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.730815 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.730784 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk"] Oct 31 01:59:20.734980 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.734950 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:20.792875 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.792834 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk"] Oct 31 01:59:20.802973 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.802944 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:20.803144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.802994 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.803144 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.803129 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df752\" (UniqueName: \"kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752\") pod \"payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:20.803272 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.803241 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzq5h\" (UniqueName: \"kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h\") pod \"payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.803711 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.803694 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.860090 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.860062 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzq5h\" (UniqueName: \"kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h\") pod \"payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.904067 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.904036 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df752\" (UniqueName: \"kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752\") pod \"payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:20.904243 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.904190 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:20.904854 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.904836 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:20.925187 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.925159 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:20.960596 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:20.960565 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df752\" (UniqueName: \"kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752\") pod \"payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:21.050350 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.050277 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:21.300733 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.300657 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997"] Oct 31 01:59:21.302034 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:59:21.301993 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d43d6f_7e13_4ab6_a1f7_faf4f219d72b.slice/crio-dbff5bd5769309c889895ad8e969d36e6893f4988cdf6e01ad879a7b34c32cf4 WatchSource:0}: Error finding container dbff5bd5769309c889895ad8e969d36e6893f4988cdf6e01ad879a7b34c32cf4: Status 404 returned error can't find the container with id dbff5bd5769309c889895ad8e969d36e6893f4988cdf6e01ad879a7b34c32cf4 Oct 31 01:59:21.424509 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.424480 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk"] Oct 31 01:59:21.425772 ip-10-0-136-249 kubenswrapper[5756]: W1031 01:59:21.425741 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27677ad2_39d3_4fa5_9fc5_bd1d602e1401.slice/crio-39cda8132aa6a4ba06714213cb63982711120cff58f37fdfe45fd5c53729c29c WatchSource:0}: Error finding container 39cda8132aa6a4ba06714213cb63982711120cff58f37fdfe45fd5c53729c29c: Status 404 returned error can't find the container with id 39cda8132aa6a4ba06714213cb63982711120cff58f37fdfe45fd5c53729c29c Oct 31 01:59:21.572193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.572106 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" event={"ID":"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b","Type":"ContainerStarted","Data":"b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b"} Oct 31 01:59:21.572193 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.572167 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" event={"ID":"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b","Type":"ContainerStarted","Data":"dbff5bd5769309c889895ad8e969d36e6893f4988cdf6e01ad879a7b34c32cf4"} Oct 31 01:59:21.572643 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.572263 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:21.573801 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.573779 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" event={"ID":"27677ad2-39d3-4fa5-9fc5-bd1d602e1401","Type":"ContainerStarted","Data":"e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66"} Oct 31 01:59:21.573958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.573811 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" event={"ID":"27677ad2-39d3-4fa5-9fc5-bd1d602e1401","Type":"ContainerStarted","Data":"39cda8132aa6a4ba06714213cb63982711120cff58f37fdfe45fd5c53729c29c"} Oct 31 01:59:21.573958 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.573898 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:21.684096 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.684044 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" podStartSLOduration=1.6840290470000001 podStartE2EDuration="1.684029047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:21.682615651 +0000 UTC m=+3733.489583205" watchObservedRunningTime="2026-10-31 01:59:21.684029047 +0000 UTC m=+3733.490996617" Oct 31 01:59:21.796967 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:21.796907 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" podStartSLOduration=1.79689093 podStartE2EDuration="1.79689093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:21.795233953 +0000 UTC m=+3733.602201499" watchObservedRunningTime="2026-10-31 01:59:21.79689093 +0000 UTC m=+3733.603858473" Oct 31 01:59:25.606490 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.606447 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" podUID="d8f121e3-3480-4349-8424-21241105353a" containerName="main" containerID="cri-o://02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7" gracePeriod=45 Oct 31 01:59:25.741151 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.741109 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" podUID="d8f121e3-3480-4349-8424-21241105353a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.214:8000/ready\": dial tcp 10.134.0.214:8000: connect: connection refused" Oct 31 01:59:25.875048 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.874986 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:59:25.966571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966539 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt\" (UniqueName: \"kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt\") pod \"d8f121e3-3480-4349-8424-21241105353a\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " Oct 31 01:59:25.966571 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966578 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home\") pod \"d8f121e3-3480-4349-8424-21241105353a\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " Oct 31 01:59:25.966780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966597 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm\") pod \"d8f121e3-3480-4349-8424-21241105353a\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " Oct 31 01:59:25.966780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966639 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir\") pod \"d8f121e3-3480-4349-8424-21241105353a\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " Oct 31 01:59:25.966780 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966730 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs\") pod \"d8f121e3-3480-4349-8424-21241105353a\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " Oct 31 01:59:25.966978 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966800 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache\") pod \"d8f121e3-3480-4349-8424-21241105353a\" (UID: \"d8f121e3-3480-4349-8424-21241105353a\") " Oct 31 01:59:25.966978 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.966889 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home" pod "d8f121e3-3480-4349-8424-21241105353a" (UID: "d8f121e3-3480-4349-8424-21241105353a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:25.967091 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.967011 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir" pod "d8f121e3-3480-4349-8424-21241105353a" (UID: "d8f121e3-3480-4349-8424-21241105353a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:25.967153 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.967115 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache" pod "d8f121e3-3480-4349-8424-21241105353a" (UID: "d8f121e3-3480-4349-8424-21241105353a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:25.967309 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.967294 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:25.967377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.967311 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:25.967377 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.967321 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:25.969029 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.969007 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm" pod "d8f121e3-3480-4349-8424-21241105353a" (UID: "d8f121e3-3480-4349-8424-21241105353a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:59:25.969133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.969055 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs" pod "d8f121e3-3480-4349-8424-21241105353a" (UID: "d8f121e3-3480-4349-8424-21241105353a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:59:25.969133 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:25.969119 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt" pod "d8f121e3-3480-4349-8424-21241105353a" (UID: "d8f121e3-3480-4349-8424-21241105353a"). InnerVolumeSpecName "kube-api-access-69jwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:59:26.068084 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.068049 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jwt\" (UniqueName: \"kubernetes.io/projected/d8f121e3-3480-4349-8424-21241105353a-kube-api-access-69jwt\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:26.068084 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.068079 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8f121e3-3480-4349-8424-21241105353a-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:26.068084 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.068089 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8f121e3-3480-4349-8424-21241105353a-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 01:59:26.607447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.607361 5756 generic.go:356] "Generic (PLEG): container finished" podID="d8f121e3-3480-4349-8424-21241105353a" containerID="02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7" exitCode=0 Oct 31 01:59:26.607447 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.607418 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" Oct 31 01:59:26.607992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.607446 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" event={"ID":"d8f121e3-3480-4349-8424-21241105353a","Type":"ContainerDied","Data":"02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7"} Oct 31 01:59:26.607992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.607497 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g" event={"ID":"d8f121e3-3480-4349-8424-21241105353a","Type":"ContainerDied","Data":"4d3270de768a02917765ffb0e9877e434dab36aa38244523f5ed509b8fe21e30"} Oct 31 01:59:26.607992 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.607521 5756 scope.go:122] "RemoveContainer" containerID="02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7" Oct 31 01:59:26.617093 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.617076 5756 scope.go:122] "RemoveContainer" containerID="02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7" Oct 31 01:59:26.617379 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:59:26.617351 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7\": container with ID starting with 02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7 not found: ID does not exist" containerID="02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7" Oct 31 01:59:26.617435 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.617391 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7"} err="failed to get container status \"02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7\": rpc error: code = NotFound desc = could not find container \"02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7\": container with ID starting with 02bc23303505576b6ced3d8b2f6ab466b70b5afc6bfd74b472aeea5c86bab8c7 not found: ID does not exist" Oct 31 01:59:26.770184 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.770154 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g"] Oct 31 01:59:26.774006 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:26.773984 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-9c9c3c/auth-test-model-9c9c3c-kserve-85bffd8749-xzv9g"] Oct 31 01:59:28.312574 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:28.312543 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8f121e3-3480-4349-8424-21241105353a" path="/var/lib/kubelet/pods/d8f121e3-3480-4349-8424-21241105353a/volumes" Oct 31 01:59:32.067050 ip-10-0-136-249 kubenswrapper[5756]: E1031 01:59:32.067015 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 01:59:32.583782 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:32.583756 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 01:59:32.583996 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:32.583823 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 01:59:51.279336 ip-10-0-136-249 kubenswrapper[5756]: I1031 01:59:51.279302 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:00:00.165201 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.165157 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94"] Oct 31 02:00:00.165856 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.165837 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d8f121e3-3480-4349-8424-21241105353a" containerName="main" Oct 31 02:00:00.165856 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.165856 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f121e3-3480-4349-8424-21241105353a" containerName="main" Oct 31 02:00:00.166015 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.166004 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f121e3-3480-4349-8424-21241105353a" containerName="main" Oct 31 02:00:00.169360 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.169341 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw"] Oct 31 02:00:00.169479 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.169460 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.173625 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.173602 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8"] Oct 31 02:00:00.173782 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.173709 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.178140 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.178116 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.212230 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.212202 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld"] Oct 31 02:00:00.216787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.216764 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk"] Oct 31 02:00:00.216966 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.216907 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.220632 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.220610 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4"] Oct 31 02:00:00.220766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.220733 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.224640 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.224620 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps"] Oct 31 02:00:00.224772 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.224747 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.229006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.228985 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56"] Oct 31 02:00:00.229129 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.229090 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.232657 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.232638 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8"] Oct 31 02:00:00.232778 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.232736 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.237250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.237232 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt"] Oct 31 02:00:00.237343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.237315 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.242062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.242040 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.258318 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.258295 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:00:00.265935 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.265910 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld"] Oct 31 02:00:00.268662 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.268638 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw"] Oct 31 02:00:00.271194 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271166 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8"] Oct 31 02:00:00.271485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271464 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.271548 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271518 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwg9m\" (UniqueName: \"kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m\") pod \"maas-api-key-cleanup-29890200-kgjk4\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.271645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271628 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.271711 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271673 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-kgjk4\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.271769 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271735 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xdt\" (UniqueName: \"kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.271830 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271788 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db5z9\" (UniqueName: \"kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.271922 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271901 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.271983 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.271946 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.272073 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272055 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxz4h\" (UniqueName: \"kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.272116 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272088 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxkb2\" (UniqueName: \"kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.272207 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272188 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.272266 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272233 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.272339 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272321 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpw8m\" (UniqueName: \"kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.272455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272438 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsf2\" (UniqueName: \"kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.272513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272478 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.272560 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.272547 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2kb2\" (UniqueName: \"kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.273804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.273782 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94"] Oct 31 02:00:00.326457 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.326139 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4"] Oct 31 02:00:00.327040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.327013 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps"] Oct 31 02:00:00.333394 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.333368 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8"] Oct 31 02:00:00.361137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.361096 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt"] Oct 31 02:00:00.377835 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.377798 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.378053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.377895 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-kgjk4\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.378053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.377947 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.378053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.377992 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xdt\" (UniqueName: \"kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.378053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378022 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db5z9\" (UniqueName: \"kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.378053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378050 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378083 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378131 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxz4h\" (UniqueName: \"kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378160 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxkb2\" (UniqueName: \"kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378227 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378255 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p892v\" (UniqueName: \"kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378294 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.378368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378331 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpw8m\" (UniqueName: \"kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378423 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsf2\" (UniqueName: \"kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378460 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378493 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2kb2\" (UniqueName: \"kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378524 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378598 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26t5f\" (UniqueName: \"kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f\") pod \"maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378632 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.378766 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378677 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwg9m\" (UniqueName: \"kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m\") pod \"maas-api-key-cleanup-29890200-kgjk4\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.379302 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.379279 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890200-kgjk4\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.379713 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.379686 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.380068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.380034 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.380472 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.380444 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.380580 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.378491 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.380745 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.380719 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.381219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.381196 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.381368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.381332 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.381368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.381362 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk"] Oct 31 02:00:00.385900 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.385854 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56"] Oct 31 02:00:00.437465 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.437387 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsf2\" (UniqueName: \"kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.437640 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.437543 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxz4h\" (UniqueName: \"kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.437640 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.437543 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2kb2\" (UniqueName: \"kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.438484 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.438086 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db5z9\" (UniqueName: \"kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.438981 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.438940 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxkb2\" (UniqueName: \"kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.439664 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.439639 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xdt\" (UniqueName: \"kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.439875 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.439838 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpw8m\" (UniqueName: \"kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.440632 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.440611 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwg9m\" (UniqueName: \"kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m\") pod \"maas-api-key-cleanup-29890200-kgjk4\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.479819 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.479783 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.480040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.479895 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p892v\" (UniqueName: \"kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.480040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.479983 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.480040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.480020 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26t5f\" (UniqueName: \"kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f\") pod \"maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.480574 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.480548 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.480664 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.480582 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.485174 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.485149 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:00.495402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.495378 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:00.501557 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.501530 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:00.535123 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.535095 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:00.542908 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.539105 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p892v\" (UniqueName: \"kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v\") pod \"maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.542908 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.539105 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26t5f\" (UniqueName: \"kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f\") pod \"maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.546020 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.545998 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:00.554127 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.553615 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:00.567643 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.567619 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:00.570587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.570388 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:00.587375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.582420 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:00.587375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.587028 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:00.885549 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.884831 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94"] Oct 31 02:00:00.888096 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:00.888056 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf706d8d1_ef11_4f75_84d2_df5ad5c77698.slice/crio-cc36a77f8e291f7e39283eabc43ded5da4279afe8d394a26244f72299ac19159 WatchSource:0}: Error finding container cc36a77f8e291f7e39283eabc43ded5da4279afe8d394a26244f72299ac19159: Status 404 returned error can't find the container with id cc36a77f8e291f7e39283eabc43ded5da4279afe8d394a26244f72299ac19159 Oct 31 02:00:00.929908 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.929877 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8"] Oct 31 02:00:00.932913 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:00.932885 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw"] Oct 31 02:00:00.958248 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:00.958218 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1b3e82_2283_46a6_b221_02ac4fa506bb.slice/crio-d08e24b0695250a0d9347a26b351e4676948aacda3ec532a339623621f7bca9b WatchSource:0}: Error finding container d08e24b0695250a0d9347a26b351e4676948aacda3ec532a339623621f7bca9b: Status 404 returned error can't find the container with id d08e24b0695250a0d9347a26b351e4676948aacda3ec532a339623621f7bca9b Oct 31 02:00:01.027479 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.027440 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk"] Oct 31 02:00:01.031337 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.030297 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e0cbbb_93e9_4ba3_aabd_ba64e52f4746.slice/crio-d775b3eef4c6b3d214bec2c51a11a087dbcd2bdf10142806f2b4507dbc895d5c WatchSource:0}: Error finding container d775b3eef4c6b3d214bec2c51a11a087dbcd2bdf10142806f2b4507dbc895d5c: Status 404 returned error can't find the container with id d775b3eef4c6b3d214bec2c51a11a087dbcd2bdf10142806f2b4507dbc895d5c Oct 31 02:00:01.073808 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.073780 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8"] Oct 31 02:00:01.092477 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.092453 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4"] Oct 31 02:00:01.132063 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.132025 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6ad7dc_d0a1_46de_adde_044f570eb295.slice/crio-e87c042c92aa82024f9a674f07fdac814d57ef1f95bd9f606c32bb5fff37399a WatchSource:0}: Error finding container e87c042c92aa82024f9a674f07fdac814d57ef1f95bd9f606c32bb5fff37399a: Status 404 returned error can't find the container with id e87c042c92aa82024f9a674f07fdac814d57ef1f95bd9f606c32bb5fff37399a Oct 31 02:00:01.133288 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.133168 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56"] Oct 31 02:00:01.133779 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.133241 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5555a21e_288a_42fd_a5b8_3efe6aedb7f7.slice/crio-26f06f487ed28aea76d7f415c6fa0dedf662948a1603750f33f12ae1f99467f6 WatchSource:0}: Error finding container 26f06f487ed28aea76d7f415c6fa0dedf662948a1603750f33f12ae1f99467f6: Status 404 returned error can't find the container with id 26f06f487ed28aea76d7f415c6fa0dedf662948a1603750f33f12ae1f99467f6 Oct 31 02:00:01.135726 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.135346 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc68799_ad1d_4d19_94bf_579da54c96e9.slice/crio-16bc434a193926c24e77e7856a4bc88ecee54c7c6a56ac946fbbc6f763397cb4 WatchSource:0}: Error finding container 16bc434a193926c24e77e7856a4bc88ecee54c7c6a56ac946fbbc6f763397cb4: Status 404 returned error can't find the container with id 16bc434a193926c24e77e7856a4bc88ecee54c7c6a56ac946fbbc6f763397cb4 Oct 31 02:00:01.136323 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.136303 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49157368_6608_4480_b050_a0bd1952308c.slice/crio-9c0ac05ab6dfabfe2e3dcdedf6cfc19fb4f50b474f65eda0a26c56fccc90c523 WatchSource:0}: Error finding container 9c0ac05ab6dfabfe2e3dcdedf6cfc19fb4f50b474f65eda0a26c56fccc90c523: Status 404 returned error can't find the container with id 9c0ac05ab6dfabfe2e3dcdedf6cfc19fb4f50b474f65eda0a26c56fccc90c523 Oct 31 02:00:01.137093 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.137073 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt"] Oct 31 02:00:01.141985 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.141966 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld"] Oct 31 02:00:01.150194 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.150170 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bed8cf_6b22_4615_993f_28d59cccb13c.slice/crio-6be356c63f4118e61219d51a6c93548b9a19882e079e5cd9148198be29900b40 WatchSource:0}: Error finding container 6be356c63f4118e61219d51a6c93548b9a19882e079e5cd9148198be29900b40: Status 404 returned error can't find the container with id 6be356c63f4118e61219d51a6c93548b9a19882e079e5cd9148198be29900b40 Oct 31 02:00:01.155248 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.155221 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps"] Oct 31 02:00:01.170005 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:01.169983 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd15fe5a_40cc_4a47_a91b_283ab2fcf75e.slice/crio-cef617de509a2cf055a9504108b06c719bc61155e64204e26c073028f8a05bf6 WatchSource:0}: Error finding container cef617de509a2cf055a9504108b06c719bc61155e64204e26c073028f8a05bf6: Status 404 returned error can't find the container with id cef617de509a2cf055a9504108b06c719bc61155e64204e26c073028f8a05bf6 Oct 31 02:00:01.842729 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.842694 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerStarted","Data":"393e50a1b08a5098e0d647b67b4b2a755c8077160eb2ae3bb7c763b1736102f4"} Oct 31 02:00:01.842951 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.842760 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerStarted","Data":"6be356c63f4118e61219d51a6c93548b9a19882e079e5cd9148198be29900b40"} Oct 31 02:00:01.844663 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.844619 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerStarted","Data":"decfd8657f2f5a4b4194d4e6326cb94ec80cc40cfd829b19fb975e6f8b0fc8ec"} Oct 31 02:00:01.844807 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.844680 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerStarted","Data":"cc36a77f8e291f7e39283eabc43ded5da4279afe8d394a26244f72299ac19159"} Oct 31 02:00:01.846352 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.846325 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerStarted","Data":"a2540fb7a62dda3bf81eb6ada7b6621c51405dbc663a22768fe3116243cdfc49"} Oct 31 02:00:01.846470 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.846374 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerStarted","Data":"26f06f487ed28aea76d7f415c6fa0dedf662948a1603750f33f12ae1f99467f6"} Oct 31 02:00:01.848031 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.847991 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerStarted","Data":"ef1d482dfbcdd0384642168b9111c7b91e0e4b40be8bffdf959c416e16909161"} Oct 31 02:00:01.848163 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.848056 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerStarted","Data":"cef617de509a2cf055a9504108b06c719bc61155e64204e26c073028f8a05bf6"} Oct 31 02:00:01.849616 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.849596 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerStarted","Data":"a98488bab182bb6686699746702b2bb91e44e01092e31a85cd972e36d94449dc"} Oct 31 02:00:01.849740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.849642 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerStarted","Data":"9c0ac05ab6dfabfe2e3dcdedf6cfc19fb4f50b474f65eda0a26c56fccc90c523"} Oct 31 02:00:01.851816 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.851796 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerStarted","Data":"9363d71004761de2705264d1c88906ada6c5a799517ee397d2ea01685ca5cf1d"} Oct 31 02:00:01.851917 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.851844 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerStarted","Data":"16bc434a193926c24e77e7856a4bc88ecee54c7c6a56ac946fbbc6f763397cb4"} Oct 31 02:00:01.853486 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.853467 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerStarted","Data":"a233200f4a44354c0213955d882f764e9bd6786577f0e77229bb5e5d997fd947"} Oct 31 02:00:01.853606 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.853501 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerStarted","Data":"d08e24b0695250a0d9347a26b351e4676948aacda3ec532a339623621f7bca9b"} Oct 31 02:00:01.855358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.855334 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerStarted","Data":"5632d8f66019139e21458ff432fdcbf6d1c533cdf3c910f62058da1553101b16"} Oct 31 02:00:01.855480 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.855379 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerStarted","Data":"08f9c2d236384494ca0d4760557b2fb3df2aae7b6095a7b3ed11eaaebd3b5e3b"} Oct 31 02:00:01.857022 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.856995 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerStarted","Data":"66dfc959e585eae04c24a6c486132afd857c4ddce941bad164ab1f89aa585185"} Oct 31 02:00:01.857135 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.857047 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerStarted","Data":"d775b3eef4c6b3d214bec2c51a11a087dbcd2bdf10142806f2b4507dbc895d5c"} Oct 31 02:00:01.858531 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.858509 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerStarted","Data":"a10d7d99cd2fa9dd785c4d45e38cac6c3f6ea2b7af75a3416f7576b5173b9c6a"} Oct 31 02:00:01.858624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.858550 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerStarted","Data":"e87c042c92aa82024f9a674f07fdac814d57ef1f95bd9f606c32bb5fff37399a"} Oct 31 02:00:01.959720 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:01.959656 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" podStartSLOduration=1.959642194 podStartE2EDuration="1.959642194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:01.95767275 +0000 UTC m=+3773.764640295" watchObservedRunningTime="2026-10-31 02:00:01.959642194 +0000 UTC m=+3773.766609761" Oct 31 02:00:02.071346 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.071293 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" podStartSLOduration=2.071279004 podStartE2EDuration="2.071279004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.068928765 +0000 UTC m=+3773.875896319" watchObservedRunningTime="2026-10-31 02:00:02.071279004 +0000 UTC m=+3773.878246546" Oct 31 02:00:02.071516 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:02.071378 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:00:02.182357 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.182311 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" podStartSLOduration=2.182296858 podStartE2EDuration="2.182296858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.179792704 +0000 UTC m=+3773.986760245" watchObservedRunningTime="2026-10-31 02:00:02.182296858 +0000 UTC m=+3773.989264398" Oct 31 02:00:02.291615 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.291559 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" podStartSLOduration=2.29154424 podStartE2EDuration="2.29154424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.290241242 +0000 UTC m=+3774.097208785" watchObservedRunningTime="2026-10-31 02:00:02.29154424 +0000 UTC m=+3774.098511783" Oct 31 02:00:02.402089 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.402031 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" podStartSLOduration=2.402016 podStartE2EDuration="2.402016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.400830873 +0000 UTC m=+3774.207798417" watchObservedRunningTime="2026-10-31 02:00:02.402016 +0000 UTC m=+3774.208983685" Oct 31 02:00:02.513745 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.513682 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" podStartSLOduration=2.513664044 podStartE2EDuration="2.513664044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.510775265 +0000 UTC m=+3774.317742809" watchObservedRunningTime="2026-10-31 02:00:02.513664044 +0000 UTC m=+3774.320631588" Oct 31 02:00:02.623046 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.622935 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" podStartSLOduration=2.62291489 podStartE2EDuration="2.62291489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.622157559 +0000 UTC m=+3774.429125127" watchObservedRunningTime="2026-10-31 02:00:02.62291489 +0000 UTC m=+3774.429882435" Oct 31 02:00:02.736439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.736382 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" podStartSLOduration=2.73636783 podStartE2EDuration="2.73636783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.733110286 +0000 UTC m=+3774.540077829" watchObservedRunningTime="2026-10-31 02:00:02.73636783 +0000 UTC m=+3774.543335423" Oct 31 02:00:02.845235 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.845182 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" podStartSLOduration=2.845167094 podStartE2EDuration="2.845167094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.843519091 +0000 UTC m=+3774.650486647" watchObservedRunningTime="2026-10-31 02:00:02.845167094 +0000 UTC m=+3774.652134669" Oct 31 02:00:02.868412 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.868383 5756 generic.go:356] "Generic (PLEG): container finished" podID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerID="66dfc959e585eae04c24a6c486132afd857c4ddce941bad164ab1f89aa585185" exitCode=60 Oct 31 02:00:02.868573 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.868455 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerDied","Data":"66dfc959e585eae04c24a6c486132afd857c4ddce941bad164ab1f89aa585185"} Oct 31 02:00:02.868788 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.868768 5756 scope.go:122] "RemoveContainer" containerID="66dfc959e585eae04c24a6c486132afd857c4ddce941bad164ab1f89aa585185" Oct 31 02:00:02.870287 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.870263 5756 generic.go:356] "Generic (PLEG): container finished" podID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerID="a10d7d99cd2fa9dd785c4d45e38cac6c3f6ea2b7af75a3416f7576b5173b9c6a" exitCode=60 Oct 31 02:00:02.870425 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.870303 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerDied","Data":"a10d7d99cd2fa9dd785c4d45e38cac6c3f6ea2b7af75a3416f7576b5173b9c6a"} Oct 31 02:00:02.870688 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.870617 5756 scope.go:122] "RemoveContainer" containerID="a10d7d99cd2fa9dd785c4d45e38cac6c3f6ea2b7af75a3416f7576b5173b9c6a" Oct 31 02:00:02.872566 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.872541 5756 generic.go:356] "Generic (PLEG): container finished" podID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerID="393e50a1b08a5098e0d647b67b4b2a755c8077160eb2ae3bb7c763b1736102f4" exitCode=60 Oct 31 02:00:02.872774 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.872738 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerDied","Data":"393e50a1b08a5098e0d647b67b4b2a755c8077160eb2ae3bb7c763b1736102f4"} Oct 31 02:00:02.872853 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.872843 5756 scope.go:122] "RemoveContainer" containerID="393e50a1b08a5098e0d647b67b4b2a755c8077160eb2ae3bb7c763b1736102f4" Oct 31 02:00:02.874472 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.874447 5756 generic.go:356] "Generic (PLEG): container finished" podID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerID="decfd8657f2f5a4b4194d4e6326cb94ec80cc40cfd829b19fb975e6f8b0fc8ec" exitCode=60 Oct 31 02:00:02.874568 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.874522 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerDied","Data":"decfd8657f2f5a4b4194d4e6326cb94ec80cc40cfd829b19fb975e6f8b0fc8ec"} Oct 31 02:00:02.874693 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.874679 5756 scope.go:122] "RemoveContainer" containerID="decfd8657f2f5a4b4194d4e6326cb94ec80cc40cfd829b19fb975e6f8b0fc8ec" Oct 31 02:00:02.876207 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.876190 5756 generic.go:356] "Generic (PLEG): container finished" podID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerID="a2540fb7a62dda3bf81eb6ada7b6621c51405dbc663a22768fe3116243cdfc49" exitCode=60 Oct 31 02:00:02.876315 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.876274 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerDied","Data":"a2540fb7a62dda3bf81eb6ada7b6621c51405dbc663a22768fe3116243cdfc49"} Oct 31 02:00:02.876478 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.876459 5756 scope.go:122] "RemoveContainer" containerID="a2540fb7a62dda3bf81eb6ada7b6621c51405dbc663a22768fe3116243cdfc49" Oct 31 02:00:02.877803 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.877785 5756 generic.go:356] "Generic (PLEG): container finished" podID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerID="ef1d482dfbcdd0384642168b9111c7b91e0e4b40be8bffdf959c416e16909161" exitCode=60 Oct 31 02:00:02.877939 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.877891 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerDied","Data":"ef1d482dfbcdd0384642168b9111c7b91e0e4b40be8bffdf959c416e16909161"} Oct 31 02:00:02.878060 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.878047 5756 scope.go:122] "RemoveContainer" containerID="ef1d482dfbcdd0384642168b9111c7b91e0e4b40be8bffdf959c416e16909161" Oct 31 02:00:02.879480 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.879463 5756 generic.go:356] "Generic (PLEG): container finished" podID="49157368-6608-4480-b050-a0bd1952308c" containerID="a98488bab182bb6686699746702b2bb91e44e01092e31a85cd972e36d94449dc" exitCode=60 Oct 31 02:00:02.879571 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.879496 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerDied","Data":"a98488bab182bb6686699746702b2bb91e44e01092e31a85cd972e36d94449dc"} Oct 31 02:00:02.879832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.879717 5756 scope.go:122] "RemoveContainer" containerID="a98488bab182bb6686699746702b2bb91e44e01092e31a85cd972e36d94449dc" Oct 31 02:00:02.881995 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.881972 5756 generic.go:356] "Generic (PLEG): container finished" podID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerID="9363d71004761de2705264d1c88906ada6c5a799517ee397d2ea01685ca5cf1d" exitCode=60 Oct 31 02:00:02.882081 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.882044 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerDied","Data":"9363d71004761de2705264d1c88906ada6c5a799517ee397d2ea01685ca5cf1d"} Oct 31 02:00:02.882264 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.882244 5756 scope.go:122] "RemoveContainer" containerID="9363d71004761de2705264d1c88906ada6c5a799517ee397d2ea01685ca5cf1d" Oct 31 02:00:02.883706 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.883687 5756 generic.go:356] "Generic (PLEG): container finished" podID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerID="a233200f4a44354c0213955d882f764e9bd6786577f0e77229bb5e5d997fd947" exitCode=60 Oct 31 02:00:02.883809 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.883745 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerDied","Data":"a233200f4a44354c0213955d882f764e9bd6786577f0e77229bb5e5d997fd947"} Oct 31 02:00:02.884036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.884018 5756 scope.go:122] "RemoveContainer" containerID="a233200f4a44354c0213955d882f764e9bd6786577f0e77229bb5e5d997fd947" Oct 31 02:00:02.885475 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.885455 5756 generic.go:356] "Generic (PLEG): container finished" podID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerID="5632d8f66019139e21458ff432fdcbf6d1c533cdf3c910f62058da1553101b16" exitCode=60 Oct 31 02:00:02.885554 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.885512 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerDied","Data":"5632d8f66019139e21458ff432fdcbf6d1c533cdf3c910f62058da1553101b16"} Oct 31 02:00:02.885671 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.885658 5756 scope.go:122] "RemoveContainer" containerID="5632d8f66019139e21458ff432fdcbf6d1c533cdf3c910f62058da1553101b16" Oct 31 02:00:02.956024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:02.955961 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" podStartSLOduration=2.955937591 podStartE2EDuration="2.955937591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:02.954723116 +0000 UTC m=+3774.761690661" watchObservedRunningTime="2026-10-31 02:00:02.955937591 +0000 UTC m=+3774.762905136" Oct 31 02:00:03.895907 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.895778 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerStarted","Data":"0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3"} Oct 31 02:00:03.897810 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.897781 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerStarted","Data":"b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3"} Oct 31 02:00:03.899803 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.899777 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerStarted","Data":"dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c"} Oct 31 02:00:03.901894 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.901854 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerStarted","Data":"b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157"} Oct 31 02:00:03.903779 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.903757 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerStarted","Data":"52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16"} Oct 31 02:00:03.905715 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.905690 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerStarted","Data":"e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8"} Oct 31 02:00:03.907704 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.907685 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerStarted","Data":"44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58"} Oct 31 02:00:03.909671 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.909652 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerStarted","Data":"8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8"} Oct 31 02:00:03.911459 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.911439 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerStarted","Data":"c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b"} Oct 31 02:00:03.913363 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:03.913342 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerStarted","Data":"5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84"} Oct 31 02:00:04.922097 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.922062 5756 generic.go:356] "Generic (PLEG): container finished" podID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerID="52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16" exitCode=60 Oct 31 02:00:04.922586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.922139 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerDied","Data":"52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16"} Oct 31 02:00:04.922586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.922235 5756 scope.go:122] "RemoveContainer" containerID="a2540fb7a62dda3bf81eb6ada7b6621c51405dbc663a22768fe3116243cdfc49" Oct 31 02:00:04.922586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.922456 5756 scope.go:122] "RemoveContainer" containerID="52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16" Oct 31 02:00:04.922811 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.922786 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-kgjk4_odh-ai-gateway-infra(5555a21e-288a-42fd-a5b8-3efe6aedb7f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" Oct 31 02:00:04.924266 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.924243 5756 generic.go:356] "Generic (PLEG): container finished" podID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerID="e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8" exitCode=60 Oct 31 02:00:04.924433 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.924300 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerDied","Data":"e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8"} Oct 31 02:00:04.924525 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.924512 5756 scope.go:122] "RemoveContainer" containerID="e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8" Oct 31 02:00:04.924786 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.924767 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps_odh-ai-gateway-infra(cd15fe5a-40cc-4a47-a91b-283ab2fcf75e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" Oct 31 02:00:04.926087 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.926065 5756 generic.go:356] "Generic (PLEG): container finished" podID="49157368-6608-4480-b050-a0bd1952308c" containerID="44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58" exitCode=60 Oct 31 02:00:04.926214 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.926131 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerDied","Data":"44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58"} Oct 31 02:00:04.926364 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.926347 5756 scope.go:122] "RemoveContainer" containerID="44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58" Oct 31 02:00:04.926629 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.926608 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56_odh-ai-gateway-infra(49157368-6608-4480-b050-a0bd1952308c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" podUID="49157368-6608-4480-b050-a0bd1952308c" Oct 31 02:00:04.927960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.927929 5756 generic.go:356] "Generic (PLEG): container finished" podID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerID="8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8" exitCode=60 Oct 31 02:00:04.928075 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.928013 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerDied","Data":"8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8"} Oct 31 02:00:04.928185 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.928171 5756 scope.go:122] "RemoveContainer" containerID="8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8" Oct 31 02:00:04.928464 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.928441 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt_odh-ai-gateway-infra(5dc68799-ad1d-4d19-94bf-579da54c96e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" Oct 31 02:00:04.929913 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.929894 5756 generic.go:356] "Generic (PLEG): container finished" podID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerID="c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b" exitCode=60 Oct 31 02:00:04.929999 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.929953 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerDied","Data":"c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b"} Oct 31 02:00:04.930130 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.930116 5756 scope.go:122] "RemoveContainer" containerID="c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b" Oct 31 02:00:04.930379 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.930357 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8_odh-ai-gateway-infra(2f1b3e82-2283-46a6-b221-02ac4fa506bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" Oct 31 02:00:04.931851 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.931831 5756 generic.go:356] "Generic (PLEG): container finished" podID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerID="5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84" exitCode=60 Oct 31 02:00:04.931983 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.931909 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerDied","Data":"5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84"} Oct 31 02:00:04.932131 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.932114 5756 scope.go:122] "RemoveContainer" containerID="5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84" Oct 31 02:00:04.932351 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.932333 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw_odh-ai-gateway-infra(ac832c0e-c186-4015-a79b-47e38fbedcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" Oct 31 02:00:04.933888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.933825 5756 generic.go:356] "Generic (PLEG): container finished" podID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerID="0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3" exitCode=60 Oct 31 02:00:04.933977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.933889 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerDied","Data":"0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3"} Oct 31 02:00:04.934044 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.934034 5756 scope.go:122] "RemoveContainer" containerID="0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3" Oct 31 02:00:04.934248 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.934228 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk_odh-ai-gateway-infra(c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" Oct 31 02:00:04.935617 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.935600 5756 generic.go:356] "Generic (PLEG): container finished" podID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerID="b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3" exitCode=60 Oct 31 02:00:04.935708 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.935696 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerDied","Data":"b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3"} Oct 31 02:00:04.935981 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.935961 5756 scope.go:122] "RemoveContainer" containerID="b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3" Oct 31 02:00:04.936196 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.936178 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8_odh-ai-gateway-infra(dc6ad7dc-d0a1-46de-adde-044f570eb295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" Oct 31 02:00:04.936942 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.936924 5756 scope.go:122] "RemoveContainer" containerID="ef1d482dfbcdd0384642168b9111c7b91e0e4b40be8bffdf959c416e16909161" Oct 31 02:00:04.937651 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.937626 5756 generic.go:356] "Generic (PLEG): container finished" podID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerID="dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c" exitCode=60 Oct 31 02:00:04.937747 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.937699 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerDied","Data":"dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c"} Oct 31 02:00:04.937929 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.937843 5756 scope.go:122] "RemoveContainer" containerID="dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c" Oct 31 02:00:04.938115 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.938090 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld_odh-ai-gateway-infra(d4bed8cf-6b22-4615-993f-28d59cccb13c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" Oct 31 02:00:04.939530 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.939512 5756 generic.go:356] "Generic (PLEG): container finished" podID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerID="b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157" exitCode=60 Oct 31 02:00:04.939611 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.939565 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerDied","Data":"b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157"} Oct 31 02:00:04.939725 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.939712 5756 scope.go:122] "RemoveContainer" containerID="b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157" Oct 31 02:00:04.939934 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:04.939915 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94_odh-ai-gateway-infra(f706d8d1-ef11-4f75-84d2-df5ad5c77698)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" Oct 31 02:00:04.950507 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.950491 5756 scope.go:122] "RemoveContainer" containerID="a98488bab182bb6686699746702b2bb91e44e01092e31a85cd972e36d94449dc" Oct 31 02:00:04.963774 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.963753 5756 scope.go:122] "RemoveContainer" containerID="9363d71004761de2705264d1c88906ada6c5a799517ee397d2ea01685ca5cf1d" Oct 31 02:00:04.976942 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.976920 5756 scope.go:122] "RemoveContainer" containerID="a233200f4a44354c0213955d882f764e9bd6786577f0e77229bb5e5d997fd947" Oct 31 02:00:04.992589 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:04.992567 5756 scope.go:122] "RemoveContainer" containerID="5632d8f66019139e21458ff432fdcbf6d1c533cdf3c910f62058da1553101b16" Oct 31 02:00:05.005147 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.005131 5756 scope.go:122] "RemoveContainer" containerID="66dfc959e585eae04c24a6c486132afd857c4ddce941bad164ab1f89aa585185" Oct 31 02:00:05.017436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.017418 5756 scope.go:122] "RemoveContainer" containerID="a10d7d99cd2fa9dd785c4d45e38cac6c3f6ea2b7af75a3416f7576b5173b9c6a" Oct 31 02:00:05.030672 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.030656 5756 scope.go:122] "RemoveContainer" containerID="393e50a1b08a5098e0d647b67b4b2a755c8077160eb2ae3bb7c763b1736102f4" Oct 31 02:00:05.043358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.043328 5756 scope.go:122] "RemoveContainer" containerID="decfd8657f2f5a4b4194d4e6326cb94ec80cc40cfd829b19fb975e6f8b0fc8ec" Oct 31 02:00:05.948476 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.948441 5756 scope.go:122] "RemoveContainer" containerID="b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3" Oct 31 02:00:05.948941 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.948765 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8_odh-ai-gateway-infra(dc6ad7dc-d0a1-46de-adde-044f570eb295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" Oct 31 02:00:05.950118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.950095 5756 scope.go:122] "RemoveContainer" containerID="dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c" Oct 31 02:00:05.950392 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.950370 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld_odh-ai-gateway-infra(d4bed8cf-6b22-4615-993f-28d59cccb13c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" Oct 31 02:00:05.951565 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.951541 5756 scope.go:122] "RemoveContainer" containerID="b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157" Oct 31 02:00:05.951809 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.951788 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94_odh-ai-gateway-infra(f706d8d1-ef11-4f75-84d2-df5ad5c77698)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" Oct 31 02:00:05.953046 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.953029 5756 scope.go:122] "RemoveContainer" containerID="52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16" Oct 31 02:00:05.953282 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.953264 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-kgjk4_odh-ai-gateway-infra(5555a21e-288a-42fd-a5b8-3efe6aedb7f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" Oct 31 02:00:05.954530 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.954514 5756 scope.go:122] "RemoveContainer" containerID="e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8" Oct 31 02:00:05.954734 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.954716 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps_odh-ai-gateway-infra(cd15fe5a-40cc-4a47-a91b-283ab2fcf75e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" Oct 31 02:00:05.956177 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.956160 5756 scope.go:122] "RemoveContainer" containerID="44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58" Oct 31 02:00:05.956344 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.956327 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56_odh-ai-gateway-infra(49157368-6608-4480-b050-a0bd1952308c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" podUID="49157368-6608-4480-b050-a0bd1952308c" Oct 31 02:00:05.957614 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.957598 5756 scope.go:122] "RemoveContainer" containerID="8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8" Oct 31 02:00:05.957806 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.957788 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt_odh-ai-gateway-infra(5dc68799-ad1d-4d19-94bf-579da54c96e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" Oct 31 02:00:05.959103 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.959086 5756 scope.go:122] "RemoveContainer" containerID="c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b" Oct 31 02:00:05.959285 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.959267 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8_odh-ai-gateway-infra(2f1b3e82-2283-46a6-b221-02ac4fa506bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" Oct 31 02:00:05.960548 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.960534 5756 scope.go:122] "RemoveContainer" containerID="5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84" Oct 31 02:00:05.960718 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.960699 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw_odh-ai-gateway-infra(ac832c0e-c186-4015-a79b-47e38fbedcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" Oct 31 02:00:05.962068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:05.962049 5756 scope.go:122] "RemoveContainer" containerID="0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3" Oct 31 02:00:05.962265 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:05.962249 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk_odh-ai-gateway-infra(c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" Oct 31 02:00:15.305089 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:15.305046 5756 scope.go:122] "RemoveContainer" containerID="e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8" Oct 31 02:00:15.305538 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:15.305204 5756 scope.go:122] "RemoveContainer" containerID="dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c" Oct 31 02:00:15.305538 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:15.305318 5756 scope.go:122] "RemoveContainer" containerID="b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157" Oct 31 02:00:16.034524 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.034492 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerStarted","Data":"505a431839a5cac25c44490b7427240ea2fa2c45b2c3cb3bafdfc8b102f634a2"} Oct 31 02:00:16.037443 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.037415 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerStarted","Data":"ec2857a895661f7fcaf8f13cd0cdae0dd137f83c02c9b2a82ef4b96264b7833a"} Oct 31 02:00:16.039570 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.039542 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerStarted","Data":"0b2f69950dbae01d52dd3d83dd66e89e689d7da0340f855625af7745366a39ba"} Oct 31 02:00:16.304845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.304761 5756 scope.go:122] "RemoveContainer" containerID="8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8" Oct 31 02:00:16.304845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.304800 5756 scope.go:122] "RemoveContainer" containerID="44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58" Oct 31 02:00:16.305077 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.304935 5756 scope.go:122] "RemoveContainer" containerID="52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16" Oct 31 02:00:16.305077 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:16.305005 5756 scope.go:122] "RemoveContainer" containerID="b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3" Oct 31 02:00:17.049560 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.049533 5756 generic.go:356] "Generic (PLEG): container finished" podID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerID="505a431839a5cac25c44490b7427240ea2fa2c45b2c3cb3bafdfc8b102f634a2" exitCode=60 Oct 31 02:00:17.049686 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.049611 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerDied","Data":"505a431839a5cac25c44490b7427240ea2fa2c45b2c3cb3bafdfc8b102f634a2"} Oct 31 02:00:17.049686 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.049666 5756 scope.go:122] "RemoveContainer" containerID="dc499db1dd1c000c1df372b7d90506e0cca71737ab17ce5dece5e1cc3aa1725c" Oct 31 02:00:17.050113 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.050089 5756 scope.go:122] "RemoveContainer" containerID="505a431839a5cac25c44490b7427240ea2fa2c45b2c3cb3bafdfc8b102f634a2" Oct 31 02:00:17.050443 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:17.050421 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld_odh-ai-gateway-infra(d4bed8cf-6b22-4615-993f-28d59cccb13c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" Oct 31 02:00:17.051631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.051612 5756 generic.go:356] "Generic (PLEG): container finished" podID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerID="ec2857a895661f7fcaf8f13cd0cdae0dd137f83c02c9b2a82ef4b96264b7833a" exitCode=60 Oct 31 02:00:17.051710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.051679 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerDied","Data":"ec2857a895661f7fcaf8f13cd0cdae0dd137f83c02c9b2a82ef4b96264b7833a"} Oct 31 02:00:17.052018 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.052001 5756 scope.go:122] "RemoveContainer" containerID="ec2857a895661f7fcaf8f13cd0cdae0dd137f83c02c9b2a82ef4b96264b7833a" Oct 31 02:00:17.052263 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:17.052238 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94_odh-ai-gateway-infra(f706d8d1-ef11-4f75-84d2-df5ad5c77698)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" Oct 31 02:00:17.054066 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.054048 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerStarted","Data":"9e53b7f91ca624dc5c21765af3fe5d1731f272748ed82db9bc3c1b3c545cff8e"} Oct 31 02:00:17.055934 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.055914 5756 generic.go:356] "Generic (PLEG): container finished" podID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerID="0b2f69950dbae01d52dd3d83dd66e89e689d7da0340f855625af7745366a39ba" exitCode=60 Oct 31 02:00:17.056027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.055981 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerDied","Data":"0b2f69950dbae01d52dd3d83dd66e89e689d7da0340f855625af7745366a39ba"} Oct 31 02:00:17.056272 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.056256 5756 scope.go:122] "RemoveContainer" containerID="0b2f69950dbae01d52dd3d83dd66e89e689d7da0340f855625af7745366a39ba" Oct 31 02:00:17.056511 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:17.056487 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps_odh-ai-gateway-infra(cd15fe5a-40cc-4a47-a91b-283ab2fcf75e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" Oct 31 02:00:17.059808 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.059787 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerStarted","Data":"caa3d357efecf441e8886da5c96408b0bd642e91951b820d088c047fee395213"} Oct 31 02:00:17.062028 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.062005 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerStarted","Data":"51de119ef633222b17158202824162e1e5b786deb786b5e74493883074284f91"} Oct 31 02:00:17.064362 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.064333 5756 scope.go:122] "RemoveContainer" containerID="b944c5df8482388c68812b31ea7992d83cb6de93dbaf49829e70abe027ed0157" Oct 31 02:00:17.083415 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.083392 5756 scope.go:122] "RemoveContainer" containerID="e9bed133670d512dab9b0dbd2e531dfe4330b316be2898bd22156421db4a87e8" Oct 31 02:00:17.158597 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.158565 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld"] Oct 31 02:00:17.268402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.268326 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94"] Oct 31 02:00:17.304946 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.304912 5756 scope.go:122] "RemoveContainer" containerID="5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84" Oct 31 02:00:17.305095 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.305055 5756 scope.go:122] "RemoveContainer" containerID="c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b" Oct 31 02:00:17.379166 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:17.379142 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps"] Oct 31 02:00:18.075568 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.075540 5756 generic.go:356] "Generic (PLEG): container finished" podID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerID="9e53b7f91ca624dc5c21765af3fe5d1731f272748ed82db9bc3c1b3c545cff8e" exitCode=60 Oct 31 02:00:18.075695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.075611 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerDied","Data":"9e53b7f91ca624dc5c21765af3fe5d1731f272748ed82db9bc3c1b3c545cff8e"} Oct 31 02:00:18.075695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.075672 5756 scope.go:122] "RemoveContainer" containerID="52e5926040f7df7ea21583bbf2662ff3f17b358e1f856e5a50e409b0b3f3be16" Oct 31 02:00:18.076089 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.076071 5756 scope.go:122] "RemoveContainer" containerID="9e53b7f91ca624dc5c21765af3fe5d1731f272748ed82db9bc3c1b3c545cff8e" Oct 31 02:00:18.076432 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:18.076406 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890200-kgjk4_odh-ai-gateway-infra(5555a21e-288a-42fd-a5b8-3efe6aedb7f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" Oct 31 02:00:18.080345 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.080327 5756 generic.go:356] "Generic (PLEG): container finished" podID="49157368-6608-4480-b050-a0bd1952308c" containerID="dfd63ee1f223048a77cffbb6b383e5d3f031a9162c8a69ca0c5a1572f8f6e1c5" exitCode=60 Oct 31 02:00:18.080463 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.080403 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerDied","Data":"dfd63ee1f223048a77cffbb6b383e5d3f031a9162c8a69ca0c5a1572f8f6e1c5"} Oct 31 02:00:18.080787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.080761 5756 scope.go:122] "RemoveContainer" containerID="dfd63ee1f223048a77cffbb6b383e5d3f031a9162c8a69ca0c5a1572f8f6e1c5" Oct 31 02:00:18.081139 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:18.081111 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56_odh-ai-gateway-infra(49157368-6608-4480-b050-a0bd1952308c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" podUID="49157368-6608-4480-b050-a0bd1952308c" Oct 31 02:00:18.082683 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.082656 5756 generic.go:356] "Generic (PLEG): container finished" podID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerID="caa3d357efecf441e8886da5c96408b0bd642e91951b820d088c047fee395213" exitCode=60 Oct 31 02:00:18.082790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.082730 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerDied","Data":"caa3d357efecf441e8886da5c96408b0bd642e91951b820d088c047fee395213"} Oct 31 02:00:18.083127 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.083104 5756 scope.go:122] "RemoveContainer" containerID="caa3d357efecf441e8886da5c96408b0bd642e91951b820d088c047fee395213" Oct 31 02:00:18.083417 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:18.083394 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt_odh-ai-gateway-infra(5dc68799-ad1d-4d19-94bf-579da54c96e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" Oct 31 02:00:18.085781 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.085753 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerStarted","Data":"f05d9d5d37aa620ee95ce135c97308c0b37db8d5f0d808a51d4d16097da0f7fe"} Oct 31 02:00:18.087984 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.087912 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerStarted","Data":"bc5b96bb9df88c23981ef84e3199f8aacfcaa363ef783154a3f74ba1bbaa780e"} Oct 31 02:00:18.091119 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.091092 5756 generic.go:356] "Generic (PLEG): container finished" podID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerID="51de119ef633222b17158202824162e1e5b786deb786b5e74493883074284f91" exitCode=60 Oct 31 02:00:18.091215 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.091137 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerDied","Data":"51de119ef633222b17158202824162e1e5b786deb786b5e74493883074284f91"} Oct 31 02:00:18.091567 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.091547 5756 scope.go:122] "RemoveContainer" containerID="51de119ef633222b17158202824162e1e5b786deb786b5e74493883074284f91" Oct 31 02:00:18.091828 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:18.091808 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8_odh-ai-gateway-infra(dc6ad7dc-d0a1-46de-adde-044f570eb295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" Oct 31 02:00:18.092295 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.092273 5756 scope.go:122] "RemoveContainer" containerID="44adc9c21539f4a238a0b5f50252b1ec9f14fcbfa5964e0508ab668f14c97e58" Oct 31 02:00:18.112064 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.111747 5756 scope.go:122] "RemoveContainer" containerID="8f9649a764e4ac32ba88d665a767eddfac0a939f2536eb839319fc48316581d8" Oct 31 02:00:18.132318 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.131987 5756 scope.go:122] "RemoveContainer" containerID="b6fc041d20fd2865d29fdaa30ca7796a927ae70e7774659bbc5bdac6c05ed7a3" Oct 31 02:00:18.281620 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.281594 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:18.303093 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.303062 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:18.334128 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.334102 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2\" (UniqueName: \"kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2\") pod \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " Oct 31 02:00:18.334267 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.334167 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca\") pod \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\" (UID: \"f706d8d1-ef11-4f75-84d2-df5ad5c77698\") " Oct 31 02:00:18.334909 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.334535 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca" pod "f706d8d1-ef11-4f75-84d2-df5ad5c77698" (UID: "f706d8d1-ef11-4f75-84d2-df5ad5c77698"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:18.334909 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.334722 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f706d8d1-ef11-4f75-84d2-df5ad5c77698-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:18.335325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.335212 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:18.336946 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.336922 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2" pod "f706d8d1-ef11-4f75-84d2-df5ad5c77698" (UID: "f706d8d1-ef11-4f75-84d2-df5ad5c77698"). InnerVolumeSpecName "kube-api-access-c2kb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:18.435450 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.435421 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2\" (UniqueName: \"kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2\") pod \"d4bed8cf-6b22-4615-993f-28d59cccb13c\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " Oct 31 02:00:18.435832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.435567 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca\") pod \"d4bed8cf-6b22-4615-993f-28d59cccb13c\" (UID: \"d4bed8cf-6b22-4615-993f-28d59cccb13c\") " Oct 31 02:00:18.435832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.435608 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h\" (UniqueName: \"kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h\") pod \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " Oct 31 02:00:18.435832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.435685 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca\") pod \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\" (UID: \"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e\") " Oct 31 02:00:18.436040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.436002 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca" pod "d4bed8cf-6b22-4615-993f-28d59cccb13c" (UID: "d4bed8cf-6b22-4615-993f-28d59cccb13c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:18.436110 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.436092 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca" pod "cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" (UID: "cd15fe5a-40cc-4a47-a91b-283ab2fcf75e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:18.436168 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.436147 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2kb2\" (UniqueName: \"kubernetes.io/projected/f706d8d1-ef11-4f75-84d2-df5ad5c77698-kube-api-access-c2kb2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:18.436168 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.436160 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4bed8cf-6b22-4615-993f-28d59cccb13c-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:18.437426 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.437403 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2" pod "d4bed8cf-6b22-4615-993f-28d59cccb13c" (UID: "d4bed8cf-6b22-4615-993f-28d59cccb13c"). InnerVolumeSpecName "kube-api-access-rdsf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:18.437832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.437811 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h" pod "cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" (UID: "cd15fe5a-40cc-4a47-a91b-283ab2fcf75e"). InnerVolumeSpecName "kube-api-access-xxz4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:18.533497 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.533415 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8"] Oct 31 02:00:18.537590 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.537565 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxz4h\" (UniqueName: \"kubernetes.io/projected/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-kube-api-access-xxz4h\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:18.537590 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.537591 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:18.537752 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.537601 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdsf2\" (UniqueName: \"kubernetes.io/projected/d4bed8cf-6b22-4615-993f-28d59cccb13c-kube-api-access-rdsf2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:18.644798 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.644763 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4"] Oct 31 02:00:18.758662 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:18.758581 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt"] Oct 31 02:00:19.102501 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.102472 5756 generic.go:356] "Generic (PLEG): container finished" podID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerID="f05d9d5d37aa620ee95ce135c97308c0b37db8d5f0d808a51d4d16097da0f7fe" exitCode=60 Oct 31 02:00:19.102818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.102546 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerDied","Data":"f05d9d5d37aa620ee95ce135c97308c0b37db8d5f0d808a51d4d16097da0f7fe"} Oct 31 02:00:19.102818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.102611 5756 scope.go:122] "RemoveContainer" containerID="c198bf367f6cf5cd9b80c897b4a82dbe41b678c6266ed734678a2271fdb83b1b" Oct 31 02:00:19.103655 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.103012 5756 scope.go:122] "RemoveContainer" containerID="f05d9d5d37aa620ee95ce135c97308c0b37db8d5f0d808a51d4d16097da0f7fe" Oct 31 02:00:19.103655 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:19.103340 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8_odh-ai-gateway-infra(2f1b3e82-2283-46a6-b221-02ac4fa506bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" Oct 31 02:00:19.104973 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.104954 5756 generic.go:356] "Generic (PLEG): container finished" podID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerID="bc5b96bb9df88c23981ef84e3199f8aacfcaa363ef783154a3f74ba1bbaa780e" exitCode=60 Oct 31 02:00:19.105067 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.105024 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerDied","Data":"bc5b96bb9df88c23981ef84e3199f8aacfcaa363ef783154a3f74ba1bbaa780e"} Oct 31 02:00:19.105397 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.105381 5756 scope.go:122] "RemoveContainer" containerID="bc5b96bb9df88c23981ef84e3199f8aacfcaa363ef783154a3f74ba1bbaa780e" Oct 31 02:00:19.105702 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:19.105682 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw_odh-ai-gateway-infra(ac832c0e-c186-4015-a79b-47e38fbedcb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" Oct 31 02:00:19.108402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.108383 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" Oct 31 02:00:19.108527 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.108421 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld" event={"ID":"d4bed8cf-6b22-4615-993f-28d59cccb13c","Type":"ContainerDied","Data":"6be356c63f4118e61219d51a6c93548b9a19882e079e5cd9148198be29900b40"} Oct 31 02:00:19.110300 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.110222 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" Oct 31 02:00:19.110300 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.110244 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94" event={"ID":"f706d8d1-ef11-4f75-84d2-df5ad5c77698","Type":"ContainerDied","Data":"cc36a77f8e291f7e39283eabc43ded5da4279afe8d394a26244f72299ac19159"} Oct 31 02:00:19.113692 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.113671 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" event={"ID":"cd15fe5a-40cc-4a47-a91b-283ab2fcf75e","Type":"ContainerDied","Data":"cef617de509a2cf055a9504108b06c719bc61155e64204e26c073028f8a05bf6"} Oct 31 02:00:19.113794 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.113750 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps" Oct 31 02:00:19.130822 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.130800 5756 scope.go:122] "RemoveContainer" containerID="5cffe012b696e81cafbed12d81a6d89a65e17fa5707437f345511ecc4b900f84" Oct 31 02:00:19.147529 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.147508 5756 scope.go:122] "RemoveContainer" containerID="505a431839a5cac25c44490b7427240ea2fa2c45b2c3cb3bafdfc8b102f634a2" Oct 31 02:00:19.161051 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.160949 5756 scope.go:122] "RemoveContainer" containerID="ec2857a895661f7fcaf8f13cd0cdae0dd137f83c02c9b2a82ef4b96264b7833a" Oct 31 02:00:19.172338 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.172213 5756 scope.go:122] "RemoveContainer" containerID="0b2f69950dbae01d52dd3d83dd66e89e689d7da0340f855625af7745366a39ba" Oct 31 02:00:19.271303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.271277 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:19.304500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.304476 5756 scope.go:122] "RemoveContainer" containerID="0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3" Oct 31 02:00:19.306636 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.306612 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56"] Oct 31 02:00:19.330414 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.330383 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:19.352059 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.352033 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f\" (UniqueName: \"kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f\") pod \"5dc68799-ad1d-4d19-94bf-579da54c96e9\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " Oct 31 02:00:19.352227 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.352095 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca\") pod \"5dc68799-ad1d-4d19-94bf-579da54c96e9\" (UID: \"5dc68799-ad1d-4d19-94bf-579da54c96e9\") " Oct 31 02:00:19.352641 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.352610 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca" pod "5dc68799-ad1d-4d19-94bf-579da54c96e9" (UID: "5dc68799-ad1d-4d19-94bf-579da54c96e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:19.354421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.354394 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f" pod "5dc68799-ad1d-4d19-94bf-579da54c96e9" (UID: "5dc68799-ad1d-4d19-94bf-579da54c96e9"). InnerVolumeSpecName "kube-api-access-26t5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:19.371311 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.371267 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:19.416809 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.416782 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw"] Oct 31 02:00:19.453720 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.453691 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca\") pod \"dc6ad7dc-d0a1-46de-adde-044f570eb295\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " Oct 31 02:00:19.454120 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.453744 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v\" (UniqueName: \"kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v\") pod \"dc6ad7dc-d0a1-46de-adde-044f570eb295\" (UID: \"dc6ad7dc-d0a1-46de-adde-044f570eb295\") " Oct 31 02:00:19.454120 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.453823 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m\" (UniqueName: \"kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m\") pod \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " Oct 31 02:00:19.454120 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.453942 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca\") pod \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\" (UID: \"5555a21e-288a-42fd-a5b8-3efe6aedb7f7\") " Oct 31 02:00:19.454291 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.454114 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca" pod "dc6ad7dc-d0a1-46de-adde-044f570eb295" (UID: "dc6ad7dc-d0a1-46de-adde-044f570eb295"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:19.454344 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.454302 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca" pod "5555a21e-288a-42fd-a5b8-3efe6aedb7f7" (UID: "5555a21e-288a-42fd-a5b8-3efe6aedb7f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:19.454483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.454463 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.454483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.454483 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26t5f\" (UniqueName: \"kubernetes.io/projected/5dc68799-ad1d-4d19-94bf-579da54c96e9-kube-api-access-26t5f\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.454629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.454493 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6ad7dc-d0a1-46de-adde-044f570eb295-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.454629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.454501 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc68799-ad1d-4d19-94bf-579da54c96e9-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.529075 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.529036 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v" pod "dc6ad7dc-d0a1-46de-adde-044f570eb295" (UID: "dc6ad7dc-d0a1-46de-adde-044f570eb295"). InnerVolumeSpecName "kube-api-access-p892v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:19.529238 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.529090 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m" pod "5555a21e-288a-42fd-a5b8-3efe6aedb7f7" (UID: "5555a21e-288a-42fd-a5b8-3efe6aedb7f7"). InnerVolumeSpecName "kube-api-access-jwg9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:19.555607 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.555577 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p892v\" (UniqueName: \"kubernetes.io/projected/dc6ad7dc-d0a1-46de-adde-044f570eb295-kube-api-access-p892v\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.555735 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.555613 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwg9m\" (UniqueName: \"kubernetes.io/projected/5555a21e-288a-42fd-a5b8-3efe6aedb7f7-kube-api-access-jwg9m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.576474 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.576449 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:19.656290 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.656265 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m\" (UniqueName: \"kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m\") pod \"49157368-6608-4480-b050-a0bd1952308c\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " Oct 31 02:00:19.656487 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.656472 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca\") pod \"49157368-6608-4480-b050-a0bd1952308c\" (UID: \"49157368-6608-4480-b050-a0bd1952308c\") " Oct 31 02:00:19.656810 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.656788 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca" pod "49157368-6608-4480-b050-a0bd1952308c" (UID: "49157368-6608-4480-b050-a0bd1952308c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:19.658329 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.658306 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m" pod "49157368-6608-4480-b050-a0bd1952308c" (UID: "49157368-6608-4480-b050-a0bd1952308c"). InnerVolumeSpecName "kube-api-access-wpw8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:19.757931 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.757895 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49157368-6608-4480-b050-a0bd1952308c-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.757931 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.757924 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpw8m\" (UniqueName: \"kubernetes.io/projected/49157368-6608-4480-b050-a0bd1952308c-kube-api-access-wpw8m\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:19.800331 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.800287 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94"] Oct 31 02:00:19.804250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.804227 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-95404c-29890200-tzb94"] Oct 31 02:00:19.865043 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.865012 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8"] Oct 31 02:00:19.911090 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.911006 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld"] Oct 31 02:00:19.914846 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:19.914818 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fc058-29890200-thqld"] Oct 31 02:00:20.022163 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.022125 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps"] Oct 31 02:00:20.026909 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.026878 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-083e17-29890200-px4ps"] Oct 31 02:00:20.124511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.124484 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" Oct 31 02:00:20.124691 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.124510 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4" event={"ID":"5555a21e-288a-42fd-a5b8-3efe6aedb7f7","Type":"ContainerDied","Data":"26f06f487ed28aea76d7f415c6fa0dedf662948a1603750f33f12ae1f99467f6"} Oct 31 02:00:20.124691 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.124595 5756 scope.go:122] "RemoveContainer" containerID="9e53b7f91ca624dc5c21765af3fe5d1731f272748ed82db9bc3c1b3c545cff8e" Oct 31 02:00:20.127273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.127241 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" Oct 31 02:00:20.127273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.127249 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56" event={"ID":"49157368-6608-4480-b050-a0bd1952308c","Type":"ContainerDied","Data":"9c0ac05ab6dfabfe2e3dcdedf6cfc19fb4f50b474f65eda0a26c56fccc90c523"} Oct 31 02:00:20.129304 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.129283 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" Oct 31 02:00:20.129511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.129337 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt" event={"ID":"5dc68799-ad1d-4d19-94bf-579da54c96e9","Type":"ContainerDied","Data":"16bc434a193926c24e77e7856a4bc88ecee54c7c6a56ac946fbbc6f763397cb4"} Oct 31 02:00:20.139937 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.139328 5756 scope.go:122] "RemoveContainer" containerID="dfd63ee1f223048a77cffbb6b383e5d3f031a9162c8a69ca0c5a1572f8f6e1c5" Oct 31 02:00:20.145740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.145671 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerStarted","Data":"125be9e29376ae12d2fa0d50bf36bf293b1a4fc3fdd0511b06bb6fdb5ed20832"} Oct 31 02:00:20.148809 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.148780 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" event={"ID":"dc6ad7dc-d0a1-46de-adde-044f570eb295","Type":"ContainerDied","Data":"e87c042c92aa82024f9a674f07fdac814d57ef1f95bd9f606c32bb5fff37399a"} Oct 31 02:00:20.148960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.148926 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8" Oct 31 02:00:20.161938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.161057 5756 scope.go:122] "RemoveContainer" containerID="caa3d357efecf441e8886da5c96408b0bd642e91951b820d088c047fee395213" Oct 31 02:00:20.184055 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.184032 5756 scope.go:122] "RemoveContainer" containerID="51de119ef633222b17158202824162e1e5b786deb786b5e74493883074284f91" Oct 31 02:00:20.323047 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.313220 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:20.323047 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.319241 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" path="/var/lib/kubelet/pods/cd15fe5a-40cc-4a47-a91b-283ab2fcf75e/volumes" Oct 31 02:00:20.323047 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.319971 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" path="/var/lib/kubelet/pods/d4bed8cf-6b22-4615-993f-28d59cccb13c/volumes" Oct 31 02:00:20.323047 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.320584 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" path="/var/lib/kubelet/pods/f706d8d1-ef11-4f75-84d2-df5ad5c77698/volumes" Oct 31 02:00:20.332521 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.332498 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:20.365350 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.365319 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca\") pod \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " Oct 31 02:00:20.365617 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.365599 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt\" (UniqueName: \"kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt\") pod \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\" (UID: \"2f1b3e82-2283-46a6-b221-02ac4fa506bb\") " Oct 31 02:00:20.365890 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.365669 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca" pod "2f1b3e82-2283-46a6-b221-02ac4fa506bb" (UID: "2f1b3e82-2283-46a6-b221-02ac4fa506bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:20.366783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.366763 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1b3e82-2283-46a6-b221-02ac4fa506bb-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:20.368042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.368020 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt" pod "2f1b3e82-2283-46a6-b221-02ac4fa506bb" (UID: "2f1b3e82-2283-46a6-b221-02ac4fa506bb"). InnerVolumeSpecName "kube-api-access-z6xdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:20.467596 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.467506 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9\" (UniqueName: \"kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9\") pod \"ac832c0e-c186-4015-a79b-47e38fbedcb7\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " Oct 31 02:00:20.468090 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.467658 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca\") pod \"ac832c0e-c186-4015-a79b-47e38fbedcb7\" (UID: \"ac832c0e-c186-4015-a79b-47e38fbedcb7\") " Oct 31 02:00:20.468090 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.468033 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca" pod "ac832c0e-c186-4015-a79b-47e38fbedcb7" (UID: "ac832c0e-c186-4015-a79b-47e38fbedcb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:20.468252 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.468229 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac832c0e-c186-4015-a79b-47e38fbedcb7-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:20.468316 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.468257 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6xdt\" (UniqueName: \"kubernetes.io/projected/2f1b3e82-2283-46a6-b221-02ac4fa506bb-kube-api-access-z6xdt\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:20.469927 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.469901 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9" pod "ac832c0e-c186-4015-a79b-47e38fbedcb7" (UID: "ac832c0e-c186-4015-a79b-47e38fbedcb7"). InnerVolumeSpecName "kube-api-access-db5z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:20.568981 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.568952 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db5z9\" (UniqueName: \"kubernetes.io/projected/ac832c0e-c186-4015-a79b-47e38fbedcb7-kube-api-access-db5z9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:20.586129 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.586100 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt"] Oct 31 02:00:20.590034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.589997 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-864622-29890200-wgtlt"] Oct 31 02:00:20.759633 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.759547 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56"] Oct 31 02:00:20.763110 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:20.763039 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b6f24b-29890200-ldh56"] Oct 31 02:00:21.095526 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.095491 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8"] Oct 31 02:00:21.097888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.097852 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e621db-29890200-hfsx8"] Oct 31 02:00:21.158044 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.158010 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" event={"ID":"2f1b3e82-2283-46a6-b221-02ac4fa506bb","Type":"ContainerDied","Data":"d08e24b0695250a0d9347a26b351e4676948aacda3ec532a339623621f7bca9b"} Oct 31 02:00:21.158229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.158079 5756 scope.go:122] "RemoveContainer" containerID="f05d9d5d37aa620ee95ce135c97308c0b37db8d5f0d808a51d4d16097da0f7fe" Oct 31 02:00:21.158229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.158024 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8" Oct 31 02:00:21.159550 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.159531 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" Oct 31 02:00:21.159650 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.159558 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw" event={"ID":"ac832c0e-c186-4015-a79b-47e38fbedcb7","Type":"ContainerDied","Data":"08f9c2d236384494ca0d4760557b2fb3df2aae7b6095a7b3ed11eaaebd3b5e3b"} Oct 31 02:00:21.161624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.161602 5756 generic.go:356] "Generic (PLEG): container finished" podID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerID="125be9e29376ae12d2fa0d50bf36bf293b1a4fc3fdd0511b06bb6fdb5ed20832" exitCode=60 Oct 31 02:00:21.161733 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.161661 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerDied","Data":"125be9e29376ae12d2fa0d50bf36bf293b1a4fc3fdd0511b06bb6fdb5ed20832"} Oct 31 02:00:21.162037 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.162017 5756 scope.go:122] "RemoveContainer" containerID="125be9e29376ae12d2fa0d50bf36bf293b1a4fc3fdd0511b06bb6fdb5ed20832" Oct 31 02:00:21.162300 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:21.162280 5756 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk_odh-ai-gateway-infra(c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" Oct 31 02:00:21.169694 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.169672 5756 scope.go:122] "RemoveContainer" containerID="bc5b96bb9df88c23981ef84e3199f8aacfcaa363ef783154a3f74ba1bbaa780e" Oct 31 02:00:21.179097 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.179079 5756 scope.go:122] "RemoveContainer" containerID="0346e2a7a7fda51fc00de954da4b9095ea2e8ae0884c6a019a9997f35b0771f3" Oct 31 02:00:21.315002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.314963 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4"] Oct 31 02:00:21.318208 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.318186 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890200-kgjk4"] Oct 31 02:00:21.424889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.424785 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8"] Oct 31 02:00:21.427357 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.427330 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-9c9c3c-29890200-ks5g8"] Oct 31 02:00:21.534285 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.534256 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw"] Oct 31 02:00:21.537449 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.537425 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-b992fe-29890200-9jpzw"] Oct 31 02:00:21.996412 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:21.996378 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk"] Oct 31 02:00:22.312995 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.312965 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" path="/var/lib/kubelet/pods/2f1b3e82-2283-46a6-b221-02ac4fa506bb/volumes" Oct 31 02:00:22.313467 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.313450 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49157368-6608-4480-b050-a0bd1952308c" path="/var/lib/kubelet/pods/49157368-6608-4480-b050-a0bd1952308c/volumes" Oct 31 02:00:22.313811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.313798 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" path="/var/lib/kubelet/pods/5555a21e-288a-42fd-a5b8-3efe6aedb7f7/volumes" Oct 31 02:00:22.314506 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.314486 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" path="/var/lib/kubelet/pods/5dc68799-ad1d-4d19-94bf-579da54c96e9/volumes" Oct 31 02:00:22.314912 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.314896 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" path="/var/lib/kubelet/pods/ac832c0e-c186-4015-a79b-47e38fbedcb7/volumes" Oct 31 02:00:22.315284 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.315268 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" path="/var/lib/kubelet/pods/dc6ad7dc-d0a1-46de-adde-044f570eb295/volumes" Oct 31 02:00:22.333198 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.333180 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:22.491244 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.491211 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2\" (UniqueName: \"kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2\") pod \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " Oct 31 02:00:22.491421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.491332 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca\") pod \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\" (UID: \"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746\") " Oct 31 02:00:22.491749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.491724 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca" pod "c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" (UID: "c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:22.493624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.493592 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2" pod "c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" (UID: "c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746"). InnerVolumeSpecName "kube-api-access-zxkb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:22.593217 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.593128 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxkb2\" (UniqueName: \"kubernetes.io/projected/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-kube-api-access-zxkb2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:22.593217 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:22.593156 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746-maas-api-service-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:23.184267 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:23.184234 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" event={"ID":"c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746","Type":"ContainerDied","Data":"d775b3eef4c6b3d214bec2c51a11a087dbcd2bdf10142806f2b4507dbc895d5c"} Oct 31 02:00:23.184267 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:23.184265 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk" Oct 31 02:00:23.184469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:23.184291 5756 scope.go:122] "RemoveContainer" containerID="125be9e29376ae12d2fa0d50bf36bf293b1a4fc3fdd0511b06bb6fdb5ed20832" Oct 31 02:00:23.346341 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:23.346303 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk"] Oct 31 02:00:23.395437 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:23.395403 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-dc9e63-29890200-bzmwk"] Oct 31 02:00:24.311333 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:24.311299 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" path="/var/lib/kubelet/pods/c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746/volumes" Oct 31 02:00:25.771144 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:25.771114 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:00:31.416018 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.415985 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7918-d973bf"] Oct 31 02:00:31.416541 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416527 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.416587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416543 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.416587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416555 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.416587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416560 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.416587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416571 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.416587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416575 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.416587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416584 5756 cpu_manager.go:414] "containerMap: removing container" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416590 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416598 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416604 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416614 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416619 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416625 5756 cpu_manager.go:414] "containerMap: removing container" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416629 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416638 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416642 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416650 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416654 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416662 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416666 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416674 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416678 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416687 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416692 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416698 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416702 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416713 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416718 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416724 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416729 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416736 5756 cpu_manager.go:414] "containerMap: removing container" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416741 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416746 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416751 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416760 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.416777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416765 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416792 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416798 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416806 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416811 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416817 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416822 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416827 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416831 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416838 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416843 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416852 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416875 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416888 5756 cpu_manager.go:414] "containerMap: removing container" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416894 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416980 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416988 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.416994 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417001 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417006 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417012 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417017 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417023 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417028 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417035 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417039 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417044 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417049 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417054 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac832c0e-c186-4015-a79b-47e38fbedcb7" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417060 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417066 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417071 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417077 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417082 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417088 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417094 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417100 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417108 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1b3e82-2283-46a6-b221-02ac4fa506bb" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417114 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.417921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.417121 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc68799-ad1d-4d19-94bf-579da54c96e9" containerName="cleanup" Oct 31 02:00:31.421790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.421774 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:31.475400 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.475370 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-d973bf"] Oct 31 02:00:31.520950 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.520917 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vxhdm\"" Oct 31 02:00:31.607197 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.607162 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrfb\" (UniqueName: \"kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb\") pod \"test-curl-7918-d973bf\" (UID: \"769867f9-435f-489d-a9df-6640f8c61db5\") " pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:31.708514 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.708487 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrfb\" (UniqueName: \"kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb\") pod \"test-curl-7918-d973bf\" (UID: \"769867f9-435f-489d-a9df-6640f8c61db5\") " pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:31.764651 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:31.764624 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrfb\" (UniqueName: \"kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb\") pod \"test-curl-7918-d973bf\" (UID: \"769867f9-435f-489d-a9df-6640f8c61db5\") " pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:32.034436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:32.034402 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:32.070797 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:32.070756 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:00:32.409541 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:32.409514 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7918-d973bf"] Oct 31 02:00:32.411304 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:32.411271 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod769867f9_435f_489d_a9df_6640f8c61db5.slice/crio-d8f3eba81b0aca834b07337a83fa8bba68eb7f0ce16fd39617e4c2ff842b0eed WatchSource:0}: Error finding container d8f3eba81b0aca834b07337a83fa8bba68eb7f0ce16fd39617e4c2ff842b0eed: Status 404 returned error can't find the container with id d8f3eba81b0aca834b07337a83fa8bba68eb7f0ce16fd39617e4c2ff842b0eed Oct 31 02:00:33.251696 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:33.251657 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-d973bf" event={"ID":"769867f9-435f-489d-a9df-6640f8c61db5","Type":"ContainerStarted","Data":"b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77"} Oct 31 02:00:33.251696 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:33.251701 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7918-d973bf" event={"ID":"769867f9-435f-489d-a9df-6640f8c61db5","Type":"ContainerStarted","Data":"d8f3eba81b0aca834b07337a83fa8bba68eb7f0ce16fd39617e4c2ff842b0eed"} Oct 31 02:00:33.363905 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:33.363833 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7918-d973bf" podStartSLOduration=2.363818962 podStartE2EDuration="2.363818962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:33.362112458 +0000 UTC m=+3805.169080003" watchObservedRunningTime="2026-10-31 02:00:33.363818962 +0000 UTC m=+3805.170786505" Oct 31 02:00:44.826256 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:44.826217 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7918-d973bf"] Oct 31 02:00:44.826773 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:44.826414 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7918-d973bf" podUID="769867f9-435f-489d-a9df-6640f8c61db5" containerName="test-curl-7918-d973bf" containerID="cri-o://b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77" gracePeriod=2 Oct 31 02:00:44.829494 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:44.829466 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7918-d973bf"] Oct 31 02:00:44.876752 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:44.876712 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-d973bf\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="769867f9-435f-489d-a9df-6640f8c61db5" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:45.113459 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.113388 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv"] Oct 31 02:00:45.113985 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.113970 5756 cpu_manager.go:414] "containerMap: removing container" podUID="769867f9-435f-489d-a9df-6640f8c61db5" containerName="test-curl-7918-d973bf" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.113987 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769867f9-435f-489d-a9df-6640f8c61db5" containerName="test-curl-7918-d973bf" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114006 5756 cpu_manager.go:414] "containerMap: removing container" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114012 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd15fe5a-40cc-4a47-a91b-283ab2fcf75e" containerName="cleanup" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114021 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114026 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4bed8cf-6b22-4615-993f-28d59cccb13c" containerName="cleanup" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114038 5756 cpu_manager.go:414] "containerMap: removing container" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:45.114042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114042 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114052 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114057 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114079 5756 cpu_manager.go:414] "containerMap: removing container" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114083 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114190 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e0cbbb-93e9-4ba3-aabd-ba64e52f4746" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114199 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f706d8d1-ef11-4f75-84d2-df5ad5c77698" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114206 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="769867f9-435f-489d-a9df-6640f8c61db5" containerName="test-curl-7918-d973bf" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114213 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="49157368-6608-4480-b050-a0bd1952308c" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114226 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6ad7dc-d0a1-46de-adde-044f570eb295" containerName="cleanup" Oct 31 02:00:45.114305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.114232 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5555a21e-288a-42fd-a5b8-3efe6aedb7f7" containerName="cleanup" Oct 31 02:00:45.117488 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.117473 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.165463 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.165429 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.165650 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.165493 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.165650 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.165607 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgr7\" (UniqueName: \"kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.170540 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.170507 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:00:45.170680 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.170566 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:00:45.173896 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.173846 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv"] Oct 31 02:00:45.267225 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.267191 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgr7\" (UniqueName: \"kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.267397 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.267343 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.267466 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.267405 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.268006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.267973 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.269969 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.269940 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.279780 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:45.279753 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-d973bf\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="769867f9-435f-489d-a9df-6640f8c61db5" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:45.323938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.323899 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgr7\" (UniqueName: \"kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7\") pod \"maas-api-revoke-sub-2a9010d8-lc9gv\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.436496 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.436425 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:45.812118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:45.812092 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv"] Oct 31 02:00:45.814271 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:45.814241 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884690f1_3600_479f_86eb_918de71ef9d5.slice/crio-a363d1779cc8d70fbc60959b1dff0fcb3d606cacf6361a5147517691539b9ec0 WatchSource:0}: Error finding container a363d1779cc8d70fbc60959b1dff0fcb3d606cacf6361a5147517691539b9ec0: Status 404 returned error can't find the container with id a363d1779cc8d70fbc60959b1dff0fcb3d606cacf6361a5147517691539b9ec0 Oct 31 02:00:46.352304 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:46.352264 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" event={"ID":"884690f1-3600-479f-86eb-918de71ef9d5","Type":"ContainerStarted","Data":"a320d401354084a922fd394581a87b0cbc9dde11fd1e113b192740f4c2aaf585"} Oct 31 02:00:46.352702 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:46.352318 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" event={"ID":"884690f1-3600-479f-86eb-918de71ef9d5","Type":"ContainerStarted","Data":"a363d1779cc8d70fbc60959b1dff0fcb3d606cacf6361a5147517691539b9ec0"} Oct 31 02:00:46.465346 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:46.465238 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" podStartSLOduration=1.465225085 podStartE2EDuration="1.465225085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:46.463139864 +0000 UTC m=+3818.270107408" watchObservedRunningTime="2026-10-31 02:00:46.465225085 +0000 UTC m=+3818.272192672" Oct 31 02:00:47.075064 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.075041 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:47.189927 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.189810 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb\" (UniqueName: \"kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb\") pod \"769867f9-435f-489d-a9df-6640f8c61db5\" (UID: \"769867f9-435f-489d-a9df-6640f8c61db5\") " Oct 31 02:00:47.192167 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.192135 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb" pod "769867f9-435f-489d-a9df-6640f8c61db5" (UID: "769867f9-435f-489d-a9df-6640f8c61db5"). InnerVolumeSpecName "kube-api-access-9rrfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:47.291118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.291085 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrfb\" (UniqueName: \"kubernetes.io/projected/769867f9-435f-489d-a9df-6640f8c61db5-kube-api-access-9rrfb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:47.360959 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.360923 5756 generic.go:356] "Generic (PLEG): container finished" podID="884690f1-3600-479f-86eb-918de71ef9d5" containerID="a320d401354084a922fd394581a87b0cbc9dde11fd1e113b192740f4c2aaf585" exitCode=0 Oct 31 02:00:47.361458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.361005 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" event={"ID":"884690f1-3600-479f-86eb-918de71ef9d5","Type":"ContainerDied","Data":"a320d401354084a922fd394581a87b0cbc9dde11fd1e113b192740f4c2aaf585"} Oct 31 02:00:47.362452 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.362432 5756 generic.go:356] "Generic (PLEG): container finished" podID="769867f9-435f-489d-a9df-6640f8c61db5" containerID="b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77" exitCode=137 Oct 31 02:00:47.362581 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.362492 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:47.362581 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.362522 5756 scope.go:122] "RemoveContainer" containerID="b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77" Oct 31 02:00:47.374807 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.374785 5756 scope.go:122] "RemoveContainer" containerID="b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77" Oct 31 02:00:47.375118 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:47.375097 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77\": container with ID starting with b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77 not found: ID does not exist" containerID="b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77" Oct 31 02:00:47.375188 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:47.375131 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77"} err="failed to get container status \"b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77\": rpc error: code = NotFound desc = could not find container \"b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77\": container with ID starting with b52ff129a61e2362d623f7737ed4d07a7ac1c10ebd7a53abf2f19378a4e6df77 not found: ID does not exist" Oct 31 02:00:47.524462 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:47.524375 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-d973bf\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="769867f9-435f-489d-a9df-6640f8c61db5" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:48.311533 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.311493 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769867f9-435f-489d-a9df-6640f8c61db5" path="/var/lib/kubelet/pods/769867f9-435f-489d-a9df-6640f8c61db5/volumes" Oct 31 02:00:48.361117 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:00:48.361078 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7918-d973bf\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="769867f9-435f-489d-a9df-6640f8c61db5" pod="opendatahub/test-curl-7918-d973bf" Oct 31 02:00:48.519803 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.519772 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:48.608129 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.608047 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca\") pod \"884690f1-3600-479f-86eb-918de71ef9d5\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " Oct 31 02:00:48.608294 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.608156 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca\") pod \"884690f1-3600-479f-86eb-918de71ef9d5\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " Oct 31 02:00:48.608294 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.608204 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7\" (UniqueName: \"kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7\") pod \"884690f1-3600-479f-86eb-918de71ef9d5\" (UID: \"884690f1-3600-479f-86eb-918de71ef9d5\") " Oct 31 02:00:48.608517 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.608493 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca" pod "884690f1-3600-479f-86eb-918de71ef9d5" (UID: "884690f1-3600-479f-86eb-918de71ef9d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:48.608818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.608800 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:48.610310 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.610290 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca" pod "884690f1-3600-479f-86eb-918de71ef9d5" (UID: "884690f1-3600-479f-86eb-918de71ef9d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:00:48.610532 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.610518 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7" pod "884690f1-3600-479f-86eb-918de71ef9d5" (UID: "884690f1-3600-479f-86eb-918de71ef9d5"). InnerVolumeSpecName "kube-api-access-zdgr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:48.710202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.710171 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/884690f1-3600-479f-86eb-918de71ef9d5-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:48.710202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:48.710196 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgr7\" (UniqueName: \"kubernetes.io/projected/884690f1-3600-479f-86eb-918de71ef9d5-kube-api-access-zdgr7\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:49.377844 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:49.377817 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" Oct 31 02:00:49.378229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:49.377819 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv" event={"ID":"884690f1-3600-479f-86eb-918de71ef9d5","Type":"ContainerDied","Data":"a363d1779cc8d70fbc60959b1dff0fcb3d606cacf6361a5147517691539b9ec0"} Oct 31 02:00:49.378229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:49.377917 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a363d1779cc8d70fbc60959b1dff0fcb3d606cacf6361a5147517691539b9ec0" Oct 31 02:00:55.898584 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:55.898540 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct"] Oct 31 02:00:55.899137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:55.899122 5756 cpu_manager.go:414] "containerMap: removing container" podUID="884690f1-3600-479f-86eb-918de71ef9d5" containerName="revoke-keys" Oct 31 02:00:55.899183 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:55.899140 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884690f1-3600-479f-86eb-918de71ef9d5" containerName="revoke-keys" Oct 31 02:00:55.899279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:55.899269 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="884690f1-3600-479f-86eb-918de71ef9d5" containerName="revoke-keys" Oct 31 02:00:55.903118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:55.903102 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:55.955144 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:55.955115 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:00:56.000489 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.000455 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6kz\" (UniqueName: \"kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.000659 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.000502 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.000659 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.000588 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.004477 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.004456 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:00:56.007232 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.007210 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct"] Oct 31 02:00:56.101589 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.101555 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.101796 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.101661 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6kz\" (UniqueName: \"kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.101796 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.101695 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.102337 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.102317 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.104211 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.104181 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.157836 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.157759 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6kz\" (UniqueName: \"kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz\") pod \"maas-api-revoke-sub-90277319-kgkct\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.217101 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.217065 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:56.596365 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:56.596337 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct"] Oct 31 02:00:56.597765 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:00:56.597735 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode959bbe6_17bb_4802_b4db_e921dc9d609a.slice/crio-21d674bf16a48ffb2d49c26e9e151794f1adc8a82c764dbd1e07c6ec9843be9a WatchSource:0}: Error finding container 21d674bf16a48ffb2d49c26e9e151794f1adc8a82c764dbd1e07c6ec9843be9a: Status 404 returned error can't find the container with id 21d674bf16a48ffb2d49c26e9e151794f1adc8a82c764dbd1e07c6ec9843be9a Oct 31 02:00:57.434921 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:57.434887 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" event={"ID":"e959bbe6-17bb-4802-b4db-e921dc9d609a","Type":"ContainerStarted","Data":"d59fa54d590218f3d793fd653902f14a62e2ea65bcdee8afe692de6216685516"} Oct 31 02:00:57.435269 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:57.434950 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" event={"ID":"e959bbe6-17bb-4802-b4db-e921dc9d609a","Type":"ContainerStarted","Data":"21d674bf16a48ffb2d49c26e9e151794f1adc8a82c764dbd1e07c6ec9843be9a"} Oct 31 02:00:57.546724 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:57.546680 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" podStartSLOduration=2.546665323 podStartE2EDuration="2.546665323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:57.54628366 +0000 UTC m=+3829.353251204" watchObservedRunningTime="2026-10-31 02:00:57.546665323 +0000 UTC m=+3829.353632866" Oct 31 02:00:58.443325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:58.443282 5756 generic.go:356] "Generic (PLEG): container finished" podID="e959bbe6-17bb-4802-b4db-e921dc9d609a" containerID="d59fa54d590218f3d793fd653902f14a62e2ea65bcdee8afe692de6216685516" exitCode=0 Oct 31 02:00:58.443723 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:58.443378 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" event={"ID":"e959bbe6-17bb-4802-b4db-e921dc9d609a","Type":"ContainerDied","Data":"d59fa54d590218f3d793fd653902f14a62e2ea65bcdee8afe692de6216685516"} Oct 31 02:00:59.162109 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.162075 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:00:59.603499 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.603475 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:00:59.641748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.641718 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca\") pod \"e959bbe6-17bb-4802-b4db-e921dc9d609a\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " Oct 31 02:00:59.641967 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.641949 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz\" (UniqueName: \"kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz\") pod \"e959bbe6-17bb-4802-b4db-e921dc9d609a\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " Oct 31 02:00:59.642016 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.641989 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca\") pod \"e959bbe6-17bb-4802-b4db-e921dc9d609a\" (UID: \"e959bbe6-17bb-4802-b4db-e921dc9d609a\") " Oct 31 02:00:59.642072 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.642051 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca" pod "e959bbe6-17bb-4802-b4db-e921dc9d609a" (UID: "e959bbe6-17bb-4802-b4db-e921dc9d609a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:00:59.642393 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.642365 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.644228 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.644198 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca" pod "e959bbe6-17bb-4802-b4db-e921dc9d609a" (UID: "e959bbe6-17bb-4802-b4db-e921dc9d609a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:00:59.644337 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.644239 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz" pod "e959bbe6-17bb-4802-b4db-e921dc9d609a" (UID: "e959bbe6-17bb-4802-b4db-e921dc9d609a"). InnerVolumeSpecName "kube-api-access-6c6kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:59.743787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.743695 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6kz\" (UniqueName: \"kubernetes.io/projected/e959bbe6-17bb-4802-b4db-e921dc9d609a-kube-api-access-6c6kz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.743787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:00:59.743727 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e959bbe6-17bb-4802-b4db-e921dc9d609a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:01:00.458910 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:00.458883 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" Oct 31 02:01:00.459083 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:00.458913 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct" event={"ID":"e959bbe6-17bb-4802-b4db-e921dc9d609a","Type":"ContainerDied","Data":"21d674bf16a48ffb2d49c26e9e151794f1adc8a82c764dbd1e07c6ec9843be9a"} Oct 31 02:01:00.459083 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:00.458963 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21d674bf16a48ffb2d49c26e9e151794f1adc8a82c764dbd1e07c6ec9843be9a" Oct 31 02:01:02.080518 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:01:02.080482 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:01:07.548690 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.548648 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4"] Oct 31 02:01:07.549651 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.549626 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e959bbe6-17bb-4802-b4db-e921dc9d609a" containerName="revoke-keys" Oct 31 02:01:07.549760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.549652 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e959bbe6-17bb-4802-b4db-e921dc9d609a" containerName="revoke-keys" Oct 31 02:01:07.549842 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.549826 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e959bbe6-17bb-4802-b4db-e921dc9d609a" containerName="revoke-keys" Oct 31 02:01:07.554097 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.554070 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.606118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.606070 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fbe06f-openshift-default-dockercfg-tx8nm\"" Oct 31 02:01:07.638938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.638893 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639123 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.638943 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639123 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639088 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639205 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639138 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639241 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639215 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639276 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639241 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639341 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639324 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639388 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875zj\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.639441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.639429 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.661022 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.660929 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4"] Oct 31 02:01:07.740666 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740629 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.740903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740681 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875zj\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.740903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740730 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.740903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740794 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.740903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740812 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.740903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740893 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741224 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740928 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741224 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.740999 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741224 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.741028 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741224 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.741147 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741446 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.741264 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741508 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.741479 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.741926 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.741897 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.742044 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.742028 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.743394 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.743367 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.744209 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.744185 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.807805 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.807701 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.811188 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.811154 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875zj\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj\") pod \"e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:07.870821 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:07.870787 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:08.314036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.314008 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4"] Oct 31 02:01:08.315480 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:01:08.315459 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8915ddc2_27bd_4d6e_a6ec_c6d318371863.slice/crio-003f1404d504827142b723353a689f3f68899eb74333c01e6dc38d0523190746 WatchSource:0}: Error finding container 003f1404d504827142b723353a689f3f68899eb74333c01e6dc38d0523190746: Status 404 returned error can't find the container with id 003f1404d504827142b723353a689f3f68899eb74333c01e6dc38d0523190746 Oct 31 02:01:08.318481 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.318445 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:08.318600 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.318523 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:08.318600 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.318568 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:08.524367 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.524320 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" event={"ID":"8915ddc2-27bd-4d6e-a6ec-c6d318371863","Type":"ContainerStarted","Data":"0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e"} Oct 31 02:01:08.524536 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.524380 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" event={"ID":"8915ddc2-27bd-4d6e-a6ec-c6d318371863","Type":"ContainerStarted","Data":"003f1404d504827142b723353a689f3f68899eb74333c01e6dc38d0523190746"} Oct 31 02:01:08.642368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.642256 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podStartSLOduration=1.6422418159999999 podStartE2EDuration="1.642241816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:08.639592772 +0000 UTC m=+3840.446560330" watchObservedRunningTime="2026-10-31 02:01:08.642241816 +0000 UTC m=+3840.449209358" Oct 31 02:01:08.871246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:08.871219 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:09.878118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:09.878030 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:10.539009 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.538979 5756 generic.go:356] "Generic (PLEG): container finished" podID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerID="285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90" exitCode=0 Oct 31 02:01:10.539210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.539052 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" event={"ID":"3e05eef2-76d3-4dec-bea6-bef86f3fdf31","Type":"ContainerDied","Data":"285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90"} Oct 31 02:01:10.539384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.539370 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:10.539651 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.539627 5756 scope.go:122] "RemoveContainer" containerID="285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90" Oct 31 02:01:10.540572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.540513 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:01:10.541184 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.541157 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:10.541304 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.541240 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:10.541304 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:10.541286 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:11.031214 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:11.031139 5756 scope.go:122] "RemoveContainer" containerID="21da9097654c096580abc5983becc58b906609c7ca035f6128bcb280d16d72be" Oct 31 02:01:11.562430 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:11.562397 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" event={"ID":"3e05eef2-76d3-4dec-bea6-bef86f3fdf31","Type":"ContainerStarted","Data":"4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f"} Oct 31 02:01:12.549888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:12.542248 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 02:01:12.549888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:12.542370 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 02:01:12.549888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:12.542426 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 02:01:13.550892 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:13.549257 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 02:01:13.550892 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:13.549336 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 02:01:14.540171 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:14.540133 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 02:01:14.540477 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:14.540447 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 02:01:15.548888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:15.548245 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 02:01:15.548888 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:15.548346 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 02:01:16.546334 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:16.546293 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 31 02:01:16.546663 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:16.546629 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 31 02:01:17.424055 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:17.424013 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 02:01:17.545006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:17.544973 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 02:01:17.603147 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:17.603122 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 02:01:25.208147 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.208110 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww"] Oct 31 02:01:25.212443 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.212426 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.268413 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.268385 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww"] Oct 31 02:01:25.291325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.291290 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67n8\" (UniqueName: \"kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8\") pod \"payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.291504 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.291335 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.313524 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.313492 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fbe06f-dockercfg-qvxrq\"" Oct 31 02:01:25.313764 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.313562 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fbe06f\"" Oct 31 02:01:25.392564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.392525 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67n8\" (UniqueName: \"kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8\") pod \"payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.392760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.392570 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.393503 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.393477 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.451536 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.451504 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67n8\" (UniqueName: \"kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8\") pod \"payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.526858 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.526784 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:25.905824 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:25.905802 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww"] Oct 31 02:01:25.907315 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:01:25.907284 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c352096_bf01_44b5_b103_87e995864264.slice/crio-3f6a6678c2533bd1c59a600631275e96acab5c16b8105872e01464a5b73b043f WatchSource:0}: Error finding container 3f6a6678c2533bd1c59a600631275e96acab5c16b8105872e01464a5b73b043f: Status 404 returned error can't find the container with id 3f6a6678c2533bd1c59a600631275e96acab5c16b8105872e01464a5b73b043f Oct 31 02:01:26.659533 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:26.659498 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" event={"ID":"7c352096-bf01-44b5-b103-87e995864264","Type":"ContainerStarted","Data":"6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc"} Oct 31 02:01:26.659533 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:26.659549 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" event={"ID":"7c352096-bf01-44b5-b103-87e995864264","Type":"ContainerStarted","Data":"3f6a6678c2533bd1c59a600631275e96acab5c16b8105872e01464a5b73b043f"} Oct 31 02:01:26.660052 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:26.659611 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:26.773561 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:26.773505 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" podStartSLOduration=1.773490621 podStartE2EDuration="1.773490621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:26.770650564 +0000 UTC m=+3858.577618108" watchObservedRunningTime="2026-10-31 02:01:26.773490621 +0000 UTC m=+3858.580458164" Oct 31 02:01:32.067167 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:01:32.067133 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:01:37.668778 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:37.668748 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:01:41.593137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.593098 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw"] Oct 31 02:01:41.597291 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.597268 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.670006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.669965 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d50516-openshift-default-dockercfg-4wgfk\"" Oct 31 02:01:41.690653 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.690615 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbnr\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.690856 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.690660 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.690856 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.690810 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.691079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.690858 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.691079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.690910 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.691079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.691005 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.691079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.691048 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.691317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.691095 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.691317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.691135 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.702259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.702227 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw"] Oct 31 02:01:41.791812 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.791778 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.791898 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.791960 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792001 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792042 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792127 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbnr\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792174 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792221 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.792303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792250 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.799504 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.792880 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.799504 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.793176 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.799504 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.795034 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.799504 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.795539 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.799504 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.795777 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.800055 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.800029 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.801470 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.801447 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.854551 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.854472 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbnr\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.877683 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.869069 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token\") pod \"e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:41.915919 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:41.915883 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:42.353378 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.353344 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw"] Oct 31 02:01:42.355443 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:01:42.355414 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2decf12d_18bb_4c12_ad66_dddeb0a47b8e.slice/crio-7b89102584ee22f3d52fc5262d3e135f0b8e94701cc22597cf81e5693392d867 WatchSource:0}: Error finding container 7b89102584ee22f3d52fc5262d3e135f0b8e94701cc22597cf81e5693392d867: Status 404 returned error can't find the container with id 7b89102584ee22f3d52fc5262d3e135f0b8e94701cc22597cf81e5693392d867 Oct 31 02:01:42.359034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.358998 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:42.359147 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.359079 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:42.359210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.359147 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:42.782137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.780277 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" event={"ID":"2decf12d-18bb-4c12-ad66-dddeb0a47b8e","Type":"ContainerStarted","Data":"c211332e90e06a26518ac5799aaa4dfc1cadd4a3ed8c77524aa655eb166b74dd"} Oct 31 02:01:42.782137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.780397 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" event={"ID":"2decf12d-18bb-4c12-ad66-dddeb0a47b8e","Type":"ContainerStarted","Data":"7b89102584ee22f3d52fc5262d3e135f0b8e94701cc22597cf81e5693392d867"} Oct 31 02:01:42.898826 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.898773 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podStartSLOduration=1.898756761 podStartE2EDuration="1.898756761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:42.897553534 +0000 UTC m=+3874.704521078" watchObservedRunningTime="2026-10-31 02:01:42.898756761 +0000 UTC m=+3874.705724491" Oct 31 02:01:42.916749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:42.916715 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:43.922322 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:43.922295 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:44.793574 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:44.793541 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:44.794437 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:44.794421 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:01:49.337561 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.337473 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx"] Oct 31 02:01:49.342352 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.342330 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.391905 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.391822 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392117 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.391984 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq622\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392117 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392029 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392146 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392311 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392298 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392371 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392341 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392426 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392372 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392456 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.392598 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.392505 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.439583 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.439555 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bbe5e0-openshift-default-dockercfg-qs4dt\"" Oct 31 02:01:49.449661 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.449535 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx"] Oct 31 02:01:49.493445 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493413 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493641 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493454 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493641 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493495 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493641 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493561 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq622\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493641 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493618 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493695 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493760 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493782 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.493911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.493799 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.494583 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.494547 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.495018 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.494971 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.495347 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.495168 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.495573 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.495521 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.495963 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.495936 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.499080 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.499052 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.499386 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.499361 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.553225 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.553191 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.553225 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.553201 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq622\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622\") pod \"e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:49.658497 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:49.658411 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:50.095792 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.095766 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx"] Oct 31 02:01:50.097179 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:01:50.097155 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9465b6_7e9b_47bc_81eb_f0fa2d778a22.slice/crio-17e8a431b0de9bade8e8aaffdf42b258ada0ca57efc863b07032442879d89291 WatchSource:0}: Error finding container 17e8a431b0de9bade8e8aaffdf42b258ada0ca57efc863b07032442879d89291: Status 404 returned error can't find the container with id 17e8a431b0de9bade8e8aaffdf42b258ada0ca57efc863b07032442879d89291 Oct 31 02:01:50.100413 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.100383 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:50.100506 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.100449 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:50.100506 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.100479 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:01:50.834303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.834271 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" event={"ID":"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22","Type":"ContainerStarted","Data":"959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40"} Oct 31 02:01:50.834714 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.834322 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" event={"ID":"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22","Type":"ContainerStarted","Data":"17e8a431b0de9bade8e8aaffdf42b258ada0ca57efc863b07032442879d89291"} Oct 31 02:01:50.951532 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:50.951480 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podStartSLOduration=1.95146416 podStartE2EDuration="1.95146416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:50.948422929 +0000 UTC m=+3882.755390473" watchObservedRunningTime="2026-10-31 02:01:50.95146416 +0000 UTC m=+3882.758431704" Oct 31 02:01:51.659512 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:51.659478 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:51.665268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:51.665243 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:51.841365 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:51.841339 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:51.842176 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:51.842155 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:01:53.908040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:01:53.908009 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:02:02.067981 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:02:02.067943 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:02:04.056371 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:04.056334 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx"] Oct 31 02:02:04.104396 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:04.104362 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22ab9915-fk7nx"] Oct 31 02:02:04.312912 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:04.312812 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a85254-6ae0-4990-894e-9ab7d091e313" path="/var/lib/kubelet/pods/b7a85254-6ae0-4990-894e-9ab7d091e313/volumes" Oct 31 02:02:15.044680 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:15.044647 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9"] Oct 31 02:02:15.093743 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:15.093715 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9957577-sxgp9"] Oct 31 02:02:16.312624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:16.312581 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb59daab-b613-4656-a88d-1fa01d43e19f" path="/var/lib/kubelet/pods/bb59daab-b613-4656-a88d-1fa01d43e19f/volumes" Oct 31 02:02:27.143508 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.143462 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2"] Oct 31 02:02:27.149503 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.149482 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.203367 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.203339 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2"] Oct 31 02:02:27.277565 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.252784 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-8a35f7-dockercfg-qxfts\"" Oct 31 02:02:27.277849 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.258030 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-8a35f7\"" Oct 31 02:02:27.284374 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.283798 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.284374 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.283883 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntgq\" (UniqueName: \"kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq\") pod \"payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.316660 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.316276 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb"] Oct 31 02:02:27.321802 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.321778 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.373272 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.373191 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb"] Oct 31 02:02:27.385490 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.385455 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.385683 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.385528 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntgq\" (UniqueName: \"kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq\") pod \"payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.386277 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.386253 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.445982 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.445917 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntgq\" (UniqueName: \"kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq\") pod \"payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.463873 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.463835 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:27.487425 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.487398 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.487609 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.487590 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvcp\" (UniqueName: \"kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp\") pod \"payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.529012 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.528979 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs"] Oct 31 02:02:27.535637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.535614 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.604887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.604422 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.604887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.604611 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvcp\" (UniqueName: \"kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp\") pod \"payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.608887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.605773 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.630302 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.629736 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs"] Oct 31 02:02:27.648896 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.646141 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-caff19-openshift-default-dockercfg-zbgrs\"" Oct 31 02:02:27.671432 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.671395 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvcp\" (UniqueName: \"kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp\") pod \"payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:27.707473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707373 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.707473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707426 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.707473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707470 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.707775 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707524 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c6p4\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.707775 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707562 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.707775 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707684 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.707775 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707751 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.708068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707798 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.708068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.707880 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808574 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808543 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808755 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808621 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808755 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808673 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808838 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808795 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808835 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808879 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.808972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.808916 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809018 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809031 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809067 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c6p4\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809102 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809390 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809279 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809687 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809662 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.809814 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.809662 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.811311 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.811280 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.811942 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.811886 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.868399 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.868373 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2"] Oct 31 02:02:27.890533 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.890502 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.891039 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.891014 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c6p4\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4\") pod \"e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:27.936207 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:27.936178 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:28.091032 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.090990 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" event={"ID":"c238dd99-399b-4dd2-b9cf-520021f63177","Type":"ContainerStarted","Data":"aa1523623f52ac5919eba2def9ddfa155204261908a024e28eb3601e0826184c"} Oct 31 02:02:28.091202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.091056 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" event={"ID":"c238dd99-399b-4dd2-b9cf-520021f63177","Type":"ContainerStarted","Data":"cefb00428058a69f294a98bfefb29218c2542f12820ef6cd4e3bac9f8076d927"} Oct 31 02:02:28.091202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.091083 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:28.161805 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.161778 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:28.204790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.204729 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" podStartSLOduration=1.204710394 podStartE2EDuration="1.204710394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:28.201693703 +0000 UTC m=+3920.008661276" watchObservedRunningTime="2026-10-31 02:02:28.204710394 +0000 UTC m=+3920.011677939" Oct 31 02:02:28.337849 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.337819 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb"] Oct 31 02:02:28.549134 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.549098 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs"] Oct 31 02:02:28.549913 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:02:28.549880 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8fdc46_dbf4_41eb_bdca_72d32e9938b1.slice/crio-90fee7b253de878449121db27edb6c115878810f7e1bf204d46dce1ddcc5f088 WatchSource:0}: Error finding container 90fee7b253de878449121db27edb6c115878810f7e1bf204d46dce1ddcc5f088: Status 404 returned error can't find the container with id 90fee7b253de878449121db27edb6c115878810f7e1bf204d46dce1ddcc5f088 Oct 31 02:02:28.553323 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.553285 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:02:28.553414 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.553374 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:02:28.553466 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:28.553420 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:02:29.099579 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.099544 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" event={"ID":"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1","Type":"ContainerStarted","Data":"0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471"} Oct 31 02:02:29.099764 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.099607 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" event={"ID":"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1","Type":"ContainerStarted","Data":"90fee7b253de878449121db27edb6c115878810f7e1bf204d46dce1ddcc5f088"} Oct 31 02:02:29.101391 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.101361 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" event={"ID":"6fa805c7-f24b-489a-9629-689a4f596721","Type":"ContainerStarted","Data":"535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446"} Oct 31 02:02:29.101519 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.101417 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" event={"ID":"6fa805c7-f24b-489a-9629-689a4f596721","Type":"ContainerStarted","Data":"d3695f3d3525a7ff6cc41818fcc13eefa91aa7d83e4f1959054f2d04ad14e57c"} Oct 31 02:02:29.101519 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.101469 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:29.162124 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.162101 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:29.167793 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.167765 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:29.217524 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.217471 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podStartSLOduration=2.217457207 podStartE2EDuration="2.217457207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:29.214212101 +0000 UTC m=+3921.021179661" watchObservedRunningTime="2026-10-31 02:02:29.217457207 +0000 UTC m=+3921.024424741" Oct 31 02:02:29.329560 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:29.329500 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" podStartSLOduration=2.329483714 podStartE2EDuration="2.329483714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:29.326593761 +0000 UTC m=+3921.133561303" watchObservedRunningTime="2026-10-31 02:02:29.329483714 +0000 UTC m=+3921.136451258" Oct 31 02:02:30.108483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:30.108448 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:30.109467 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:30.109450 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:02:32.066904 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:02:32.066850 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:02:39.105598 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:39.105568 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:02:40.112266 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:40.112240 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:02:54.556653 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:02:54.556620 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:03:02.067896 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:03:02.067836 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:03:11.138557 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:11.138525 5756 scope.go:122] "RemoveContainer" containerID="fb55fba57926bd3a0748484ec1c8abf86581a9361ebd865ca16dad541e4cf70f" Oct 31 02:03:23.047472 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:23.047438 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx"] Oct 31 02:03:23.050154 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:23.050130 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7968cb5-2nbtx"] Oct 31 02:03:24.312534 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.312501 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d1073b-a0d1-4c04-b313-39495ea53b45" path="/var/lib/kubelet/pods/c0d1073b-a0d1-4c04-b313-39495ea53b45/volumes" Oct 31 02:03:24.502134 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.502101 5756 generic.go:356] "Generic (PLEG): container finished" podID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerID="0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e" exitCode=0 Oct 31 02:03:24.502307 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.502173 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" event={"ID":"8915ddc2-27bd-4d6e-a6ec-c6d318371863","Type":"ContainerDied","Data":"0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e"} Oct 31 02:03:24.502646 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.502629 5756 scope.go:122] "RemoveContainer" containerID="0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e" Oct 31 02:03:24.504411 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.504381 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:03:24.504528 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.504450 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:03:24.504528 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:24.504480 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:03:25.047267 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.047219 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv"] Oct 31 02:03:25.053648 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.053580 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-584140cf-phjnv"] Oct 31 02:03:25.512365 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.512286 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" event={"ID":"8915ddc2-27bd-4d6e-a6ec-c6d318371863","Type":"ContainerStarted","Data":"002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89"} Oct 31 02:03:25.546886 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.544958 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:03:25.871159 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.871130 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:03:25.872630 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.872604 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 02:03:25.872760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:25.872661 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 02:03:26.313028 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:26.312988 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1b753f-180a-4e02-bc5b-aaad131b7dda" path="/var/lib/kubelet/pods/9f1b753f-180a-4e02-bc5b-aaad131b7dda/volumes" Oct 31 02:03:26.871772 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:26.871741 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 02:03:26.872233 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:26.871789 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 02:03:27.871240 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:27.871199 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 02:03:27.871426 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:27.871273 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 02:03:28.871077 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:28.871045 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 02:03:28.871487 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:28.871106 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 02:03:29.871389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:29.871357 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 31 02:03:29.871818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:29.871424 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 31 02:03:30.876997 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:30.876970 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:03:30.878198 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:30.878171 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:03:32.066382 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:03:32.066344 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:03:32.599024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:32.598985 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:03:33.047839 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:33.047810 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m"] Oct 31 02:03:33.097245 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:33.097210 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-581f4412-b8p9m"] Oct 31 02:03:34.311950 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:34.311908 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0550d329-d52c-4f04-96bd-a3956205046b" path="/var/lib/kubelet/pods/0550d329-d52c-4f04-96bd-a3956205046b/volumes" Oct 31 02:03:35.044771 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:35.044739 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds"] Oct 31 02:03:35.093990 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:35.093962 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5b671f-7p2ds"] Oct 31 02:03:36.311992 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:03:36.311957 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4b519e-d61f-45ba-bbba-635c9579b891" path="/var/lib/kubelet/pods/3f4b519e-d61f-45ba-bbba-635c9579b891/volumes" Oct 31 02:04:02.104522 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:04:02.104476 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:04:06.843540 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.843507 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 02:04:06.844068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.843575 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 02:04:06.872059 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.872031 5756 generic.go:356] "Generic (PLEG): container finished" podID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerID="c211332e90e06a26518ac5799aaa4dfc1cadd4a3ed8c77524aa655eb166b74dd" exitCode=0 Oct 31 02:04:06.872219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.872111 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" event={"ID":"2decf12d-18bb-4c12-ad66-dddeb0a47b8e","Type":"ContainerDied","Data":"c211332e90e06a26518ac5799aaa4dfc1cadd4a3ed8c77524aa655eb166b74dd"} Oct 31 02:04:06.872535 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.872517 5756 scope.go:122] "RemoveContainer" containerID="c211332e90e06a26518ac5799aaa4dfc1cadd4a3ed8c77524aa655eb166b74dd" Oct 31 02:04:06.873758 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.873737 5756 generic.go:356] "Generic (PLEG): container finished" podID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerID="959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40" exitCode=0 Oct 31 02:04:06.873876 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.873811 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" event={"ID":"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22","Type":"ContainerDied","Data":"959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40"} Oct 31 02:04:06.874159 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.874146 5756 scope.go:122] "RemoveContainer" containerID="959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40" Oct 31 02:04:06.874398 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.874258 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:06.874398 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.874344 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:06.874398 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.874378 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:06.883234 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.876114 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:06.883234 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.876315 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:06.883234 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:06.876416 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:07.884640 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:07.884606 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" event={"ID":"2decf12d-18bb-4c12-ad66-dddeb0a47b8e","Type":"ContainerStarted","Data":"15728789ed4815da014b7ccabd709ead00d9c57b062f7455b672299ac29c6d7b"} Oct 31 02:04:07.887022 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:07.886997 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" event={"ID":"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22","Type":"ContainerStarted","Data":"93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9"} Oct 31 02:04:07.916887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:07.916272 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:04:07.916887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:07.916403 5756 patch_prober.go:29] interesting pod/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 02:04:07.916887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:07.916446 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 02:04:08.668040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:08.659257 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:04:08.668040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:08.659390 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 02:04:08.668040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:08.659436 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 02:04:08.916818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:08.916767 5756 patch_prober.go:29] interesting pod/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 02:04:08.917386 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:08.916827 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 02:04:09.662631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:09.662508 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 02:04:09.662631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:09.662581 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 02:04:09.916465 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:09.916381 5756 patch_prober.go:29] interesting pod/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 02:04:09.916465 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:09.916439 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 02:04:10.661673 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:10.661237 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 02:04:10.661673 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:10.661298 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 02:04:10.917137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:10.917043 5756 patch_prober.go:29] interesting pod/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 02:04:10.917137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:10.917106 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 02:04:11.188641 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:11.188438 5756 scope.go:122] "RemoveContainer" containerID="9ceca2d895a3d50b9d1cc0d1a2614951a88786f8c81ffbc8b13d973841576daf" Oct 31 02:04:11.659291 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:11.659256 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 02:04:11.659626 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:11.659594 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 02:04:11.919404 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:11.919319 5756 patch_prober.go:29] interesting pod/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 02:04:11.920031 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:11.919989 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 02:04:12.659889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:12.659377 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 31 02:04:12.659889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:12.659451 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 31 02:04:12.916448 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:12.916364 5756 patch_prober.go:29] interesting pod/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 31 02:04:12.916448 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:12.916428 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 31 02:04:13.664406 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:13.664377 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:04:13.664889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:13.664854 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:04:13.665676 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:13.665654 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:04:13.921842 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:13.921765 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:04:13.922179 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:13.922160 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:04:13.922943 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:13.922926 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:04:25.244992 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.244957 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz"] Oct 31 02:04:25.249479 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.249457 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.267250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.267223 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnhpr\" (UniqueName: \"kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.267380 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.267279 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.267440 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.267417 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.350527 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.350492 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:04:25.350683 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.350538 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:04:25.354524 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.354502 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz"] Oct 31 02:04:25.368786 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.368756 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnhpr\" (UniqueName: \"kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.368984 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.368805 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.368984 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.368845 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.369463 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.369443 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.371400 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.371377 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.425141 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.425105 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnhpr\" (UniqueName: \"kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr\") pod \"maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.563436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.563407 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:25.936765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:25.936736 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz"] Oct 31 02:04:25.938138 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:04:25.938108 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9220da3_2136_43c4_be84_3b0448ad0dd9.slice/crio-78eb8f1df9ae6c4503e0490036825158501d1ea0bea15dd805cf83500f092dfb WatchSource:0}: Error finding container 78eb8f1df9ae6c4503e0490036825158501d1ea0bea15dd805cf83500f092dfb: Status 404 returned error can't find the container with id 78eb8f1df9ae6c4503e0490036825158501d1ea0bea15dd805cf83500f092dfb Oct 31 02:04:26.014833 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:26.014801 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" event={"ID":"c9220da3-2136-43c4-be84-3b0448ad0dd9","Type":"ContainerStarted","Data":"78eb8f1df9ae6c4503e0490036825158501d1ea0bea15dd805cf83500f092dfb"} Oct 31 02:04:27.024305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:27.024266 5756 generic.go:356] "Generic (PLEG): container finished" podID="c9220da3-2136-43c4-be84-3b0448ad0dd9" containerID="ded4968e1bf496379174cc829d9d3d1cf809dceea6fa16a69fd5ec216132f188" exitCode=0 Oct 31 02:04:27.024768 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:27.024357 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" event={"ID":"c9220da3-2136-43c4-be84-3b0448ad0dd9","Type":"ContainerDied","Data":"ded4968e1bf496379174cc829d9d3d1cf809dceea6fa16a69fd5ec216132f188"} Oct 31 02:04:28.183151 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.183128 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:28.204510 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.204481 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca\") pod \"c9220da3-2136-43c4-be84-3b0448ad0dd9\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " Oct 31 02:04:28.204669 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.204531 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr\" (UniqueName: \"kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr\") pod \"c9220da3-2136-43c4-be84-3b0448ad0dd9\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " Oct 31 02:04:28.204669 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.204663 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca\") pod \"c9220da3-2136-43c4-be84-3b0448ad0dd9\" (UID: \"c9220da3-2136-43c4-be84-3b0448ad0dd9\") " Oct 31 02:04:28.205096 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.205064 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca" pod "c9220da3-2136-43c4-be84-3b0448ad0dd9" (UID: "c9220da3-2136-43c4-be84-3b0448ad0dd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:04:28.206783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.206759 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca" pod "c9220da3-2136-43c4-be84-3b0448ad0dd9" (UID: "c9220da3-2136-43c4-be84-3b0448ad0dd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:04:28.206889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.206774 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr" pod "c9220da3-2136-43c4-be84-3b0448ad0dd9" (UID: "c9220da3-2136-43c4-be84-3b0448ad0dd9"). InnerVolumeSpecName "kube-api-access-wnhpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:04:28.306436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.306356 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnhpr\" (UniqueName: \"kubernetes.io/projected/c9220da3-2136-43c4-be84-3b0448ad0dd9-kube-api-access-wnhpr\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:04:28.306436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.306392 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:04:28.306436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:28.306406 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9220da3-2136-43c4-be84-3b0448ad0dd9-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:04:29.040392 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:29.040367 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" Oct 31 02:04:29.040580 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:29.040360 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz" event={"ID":"c9220da3-2136-43c4-be84-3b0448ad0dd9","Type":"ContainerDied","Data":"78eb8f1df9ae6c4503e0490036825158501d1ea0bea15dd805cf83500f092dfb"} Oct 31 02:04:29.040580 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:29.040487 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78eb8f1df9ae6c4503e0490036825158501d1ea0bea15dd805cf83500f092dfb" Oct 31 02:04:32.068491 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:04:32.068454 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:04:33.532763 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:33.532729 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:04:45.110368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.110335 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 02:04:45.110788 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.110401 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 02:04:45.149845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.149814 5756 generic.go:356] "Generic (PLEG): container finished" podID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerID="0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471" exitCode=0 Oct 31 02:04:45.150020 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.149894 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" event={"ID":"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1","Type":"ContainerDied","Data":"0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471"} Oct 31 02:04:45.150336 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.150321 5756 scope.go:122] "RemoveContainer" containerID="0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471" Oct 31 02:04:45.152029 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.151999 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:45.152108 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.152070 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:45.152146 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:45.152115 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:04:46.172070 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:46.172041 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" event={"ID":"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1","Type":"ContainerStarted","Data":"fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100"} Oct 31 02:04:47.166658 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:47.166620 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:04:47.167127 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:47.166972 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 02:04:47.167258 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:47.167172 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 02:04:48.162402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:48.162366 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 02:04:48.162808 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:48.162424 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 02:04:49.162980 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:49.162944 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 02:04:49.163396 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:49.163002 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 02:04:50.162310 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:50.162270 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 02:04:50.162522 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:50.162348 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 02:04:51.162735 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:51.162698 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 31 02:04:51.163261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:51.162773 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 31 02:04:52.167792 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:52.167761 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:04:52.168257 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:52.168176 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:04:52.168951 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:52.168927 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:04:58.834429 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.834397 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc"] Oct 31 02:04:58.835048 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.835031 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c9220da3-2136-43c4-be84-3b0448ad0dd9" containerName="revoke-keys" Oct 31 02:04:58.835048 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.835048 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9220da3-2136-43c4-be84-3b0448ad0dd9" containerName="revoke-keys" Oct 31 02:04:58.835192 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.835182 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9220da3-2136-43c4-be84-3b0448ad0dd9" containerName="revoke-keys" Oct 31 02:04:58.838207 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.838187 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:58.889833 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.889809 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:04:58.913272 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.913240 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlq4f\" (UniqueName: \"kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:58.913446 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.913301 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:58.913511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.913489 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:58.939286 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.939259 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:04:58.942102 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:58.942078 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc"] Oct 31 02:04:59.015091 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.015059 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlq4f\" (UniqueName: \"kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.015260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.015105 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.015260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.015203 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.015877 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.015846 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.017609 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.017591 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.072272 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.072239 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlq4f\" (UniqueName: \"kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f\") pod \"maas-api-revoke-keys-e2e-autores-d50516-h5qlc\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.154286 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.154195 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:04:59.528513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:04:59.528479 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc"] Oct 31 02:04:59.529707 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:04:59.529678 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f498b8_91fe_4142_88e9_4d7456162fc8.slice/crio-78905f1a901aa74c92aacd454880c9ef55ec67da72eec49f1c4be91efe053d9d WatchSource:0}: Error finding container 78905f1a901aa74c92aacd454880c9ef55ec67da72eec49f1c4be91efe053d9d: Status 404 returned error can't find the container with id 78905f1a901aa74c92aacd454880c9ef55ec67da72eec49f1c4be91efe053d9d Oct 31 02:05:00.280480 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:00.280401 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" event={"ID":"57f498b8-91fe-4142-88e9-4d7456162fc8","Type":"ContainerStarted","Data":"ecc848a1b4f0a563e9cafc0b5f22197e0b9f04ad7d807f244beb632bf593cc0f"} Oct 31 02:05:00.280480 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:00.280454 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" event={"ID":"57f498b8-91fe-4142-88e9-4d7456162fc8","Type":"ContainerStarted","Data":"78905f1a901aa74c92aacd454880c9ef55ec67da72eec49f1c4be91efe053d9d"} Oct 31 02:05:00.394642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:00.394591 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" podStartSLOduration=2.394576471 podStartE2EDuration="2.394576471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:00.39242264 +0000 UTC m=+4072.199390185" watchObservedRunningTime="2026-10-31 02:05:00.394576471 +0000 UTC m=+4072.201544013" Oct 31 02:05:01.288942 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:01.288908 5756 generic.go:356] "Generic (PLEG): container finished" podID="57f498b8-91fe-4142-88e9-4d7456162fc8" containerID="ecc848a1b4f0a563e9cafc0b5f22197e0b9f04ad7d807f244beb632bf593cc0f" exitCode=0 Oct 31 02:05:01.289337 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:01.288954 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" event={"ID":"57f498b8-91fe-4142-88e9-4d7456162fc8","Type":"ContainerDied","Data":"ecc848a1b4f0a563e9cafc0b5f22197e0b9f04ad7d807f244beb632bf593cc0f"} Oct 31 02:05:02.068076 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:02.068034 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:05:02.452101 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.452076 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:05:02.564096 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.564065 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f\" (UniqueName: \"kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f\") pod \"57f498b8-91fe-4142-88e9-4d7456162fc8\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " Oct 31 02:05:02.564282 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.564106 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca\") pod \"57f498b8-91fe-4142-88e9-4d7456162fc8\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " Oct 31 02:05:02.564282 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.564262 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca\") pod \"57f498b8-91fe-4142-88e9-4d7456162fc8\" (UID: \"57f498b8-91fe-4142-88e9-4d7456162fc8\") " Oct 31 02:05:02.564645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.564619 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca" pod "57f498b8-91fe-4142-88e9-4d7456162fc8" (UID: "57f498b8-91fe-4142-88e9-4d7456162fc8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:02.564770 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.564754 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:02.566918 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.566888 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f" pod "57f498b8-91fe-4142-88e9-4d7456162fc8" (UID: "57f498b8-91fe-4142-88e9-4d7456162fc8"). InnerVolumeSpecName "kube-api-access-nlq4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:02.568464 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.568426 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca" pod "57f498b8-91fe-4142-88e9-4d7456162fc8" (UID: "57f498b8-91fe-4142-88e9-4d7456162fc8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:05:02.665872 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.665780 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlq4f\" (UniqueName: \"kubernetes.io/projected/57f498b8-91fe-4142-88e9-4d7456162fc8-kube-api-access-nlq4f\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:02.665872 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:02.665809 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f498b8-91fe-4142-88e9-4d7456162fc8-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:03.075490 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.075446 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt"] Oct 31 02:05:03.076187 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.076161 5756 cpu_manager.go:414] "containerMap: removing container" podUID="57f498b8-91fe-4142-88e9-4d7456162fc8" containerName="revoke-keys" Oct 31 02:05:03.076187 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.076183 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f498b8-91fe-4142-88e9-4d7456162fc8" containerName="revoke-keys" Oct 31 02:05:03.076386 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.076278 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f498b8-91fe-4142-88e9-4d7456162fc8" containerName="revoke-keys" Oct 31 02:05:03.078657 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.078635 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.136655 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.136624 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt"] Oct 31 02:05:03.172240 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.172204 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.172446 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.172363 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.172508 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.172470 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzbs\" (UniqueName: \"kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.273911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.273880 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.274091 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.273940 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzbs\" (UniqueName: \"kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.274091 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.274044 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.274623 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.274598 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.276336 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.276319 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.303575 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.303551 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" Oct 31 02:05:03.303705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.303589 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc" event={"ID":"57f498b8-91fe-4142-88e9-4d7456162fc8","Type":"ContainerDied","Data":"78905f1a901aa74c92aacd454880c9ef55ec67da72eec49f1c4be91efe053d9d"} Oct 31 02:05:03.303705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.303637 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78905f1a901aa74c92aacd454880c9ef55ec67da72eec49f1c4be91efe053d9d" Oct 31 02:05:03.330599 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.330538 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzbs\" (UniqueName: \"kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs\") pod \"maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.392892 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.392852 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:03.783043 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:05:03.783016 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb66553_b7b7_43b3_b3de_32f108db5558.slice/crio-ce545095d6f45f33406249392365d0524016660566fd5395edb33ad18e45cbab WatchSource:0}: Error finding container ce545095d6f45f33406249392365d0524016660566fd5395edb33ad18e45cbab: Status 404 returned error can't find the container with id ce545095d6f45f33406249392365d0524016660566fd5395edb33ad18e45cbab Oct 31 02:05:03.783499 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:03.783230 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt"] Oct 31 02:05:04.312611 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:04.312578 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" event={"ID":"bcb66553-b7b7-43b3-b3de-32f108db5558","Type":"ContainerStarted","Data":"b56311293cdb5aa3aff32f731ca68761f56ea14b6c7b3b36ee346add9b27b487"} Oct 31 02:05:04.312775 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:04.312622 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" event={"ID":"bcb66553-b7b7-43b3-b3de-32f108db5558","Type":"ContainerStarted","Data":"ce545095d6f45f33406249392365d0524016660566fd5395edb33ad18e45cbab"} Oct 31 02:05:04.425568 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:04.425450 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" podStartSLOduration=1.425432419 podStartE2EDuration="1.425432419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:04.42456588 +0000 UTC m=+4076.231533424" watchObservedRunningTime="2026-10-31 02:05:04.425432419 +0000 UTC m=+4076.232399962" Oct 31 02:05:05.320334 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:05.320296 5756 generic.go:356] "Generic (PLEG): container finished" podID="bcb66553-b7b7-43b3-b3de-32f108db5558" containerID="b56311293cdb5aa3aff32f731ca68761f56ea14b6c7b3b36ee346add9b27b487" exitCode=0 Oct 31 02:05:05.320729 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:05.320360 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" event={"ID":"bcb66553-b7b7-43b3-b3de-32f108db5558","Type":"ContainerDied","Data":"b56311293cdb5aa3aff32f731ca68761f56ea14b6c7b3b36ee346add9b27b487"} Oct 31 02:05:06.488522 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.488492 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:06.614451 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.614373 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca\") pod \"bcb66553-b7b7-43b3-b3de-32f108db5558\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " Oct 31 02:05:06.614451 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.614431 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca\") pod \"bcb66553-b7b7-43b3-b3de-32f108db5558\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " Oct 31 02:05:06.614695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.614589 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs\" (UniqueName: \"kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs\") pod \"bcb66553-b7b7-43b3-b3de-32f108db5558\" (UID: \"bcb66553-b7b7-43b3-b3de-32f108db5558\") " Oct 31 02:05:06.614777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.614750 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca" pod "bcb66553-b7b7-43b3-b3de-32f108db5558" (UID: "bcb66553-b7b7-43b3-b3de-32f108db5558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:06.615221 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.615148 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:06.616739 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.616712 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca" pod "bcb66553-b7b7-43b3-b3de-32f108db5558" (UID: "bcb66553-b7b7-43b3-b3de-32f108db5558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:05:06.616832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.616816 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs" pod "bcb66553-b7b7-43b3-b3de-32f108db5558" (UID: "bcb66553-b7b7-43b3-b3de-32f108db5558"). InnerVolumeSpecName "kube-api-access-xxzbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:06.716410 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.716378 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzbs\" (UniqueName: \"kubernetes.io/projected/bcb66553-b7b7-43b3-b3de-32f108db5558-kube-api-access-xxzbs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:06.716410 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:06.716409 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb66553-b7b7-43b3-b3de-32f108db5558-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:07.335196 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:07.335168 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" Oct 31 02:05:07.335399 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:07.335216 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt" event={"ID":"bcb66553-b7b7-43b3-b3de-32f108db5558","Type":"ContainerDied","Data":"ce545095d6f45f33406249392365d0524016660566fd5395edb33ad18e45cbab"} Oct 31 02:05:07.335399 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:07.335270 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce545095d6f45f33406249392365d0524016660566fd5395edb33ad18e45cbab" Oct 31 02:05:11.218654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:11.218624 5756 scope.go:122] "RemoveContainer" containerID="176451f867fa22bd4689545a6a93a8ecadb8a6422a3737ed94336e7d29208a8d" Oct 31 02:05:11.229306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:11.229287 5756 scope.go:122] "RemoveContainer" containerID="37eb6f98871005d0c78c09d78efc8250da73bf238e66db38500663bd5853098b" Oct 31 02:05:11.242555 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:11.242535 5756 scope.go:122] "RemoveContainer" containerID="bd5dcf3f77f6a49ecf7480101b9659ec049a5acd9c7fd9456a7e78ab0c042653" Oct 31 02:05:11.252929 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:11.252906 5756 scope.go:122] "RemoveContainer" containerID="e62d765ecf2ad3ae1314c7a0838fa57966b018b6416d11e3b45f9059934a9a0a" Oct 31 02:05:15.136501 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:15.136471 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw"] Oct 31 02:05:15.240609 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:15.240570 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr"] Oct 31 02:05:20.137428 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.137388 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" podUID="6aa99ba2-4df3-41b0-92d0-92d445980db2" containerName="payload-processing" containerID="cri-o://e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86" gracePeriod=25 Oct 31 02:05:20.241264 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.241219 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" podUID="006578cd-b9d3-405e-a323-4ae79d19a5ee" containerName="payload-pre-processing" containerID="cri-o://84a8fa74aa8356fd78e53fb22655bbbccf8f4402703e638b4faea92d1bf8c820" gracePeriod=25 Oct 31 02:05:20.405892 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.405803 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 02:05:20.423009 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.422976 5756 generic.go:356] "Generic (PLEG): container finished" podID="6aa99ba2-4df3-41b0-92d0-92d445980db2" containerID="e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86" exitCode=0 Oct 31 02:05:20.423178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.423023 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" Oct 31 02:05:20.423178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.423065 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" event={"ID":"6aa99ba2-4df3-41b0-92d0-92d445980db2","Type":"ContainerDied","Data":"e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86"} Oct 31 02:05:20.423178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.423126 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw" event={"ID":"6aa99ba2-4df3-41b0-92d0-92d445980db2","Type":"ContainerDied","Data":"5a3d5911a132fca0440f7a481e94ed2de866081ad076d3a879822d6b2ab5beb4"} Oct 31 02:05:20.423178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.423148 5756 scope.go:122] "RemoveContainer" containerID="e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86" Oct 31 02:05:20.425105 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.425083 5756 generic.go:356] "Generic (PLEG): container finished" podID="006578cd-b9d3-405e-a323-4ae79d19a5ee" containerID="84a8fa74aa8356fd78e53fb22655bbbccf8f4402703e638b4faea92d1bf8c820" exitCode=0 Oct 31 02:05:20.425245 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.425134 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" event={"ID":"006578cd-b9d3-405e-a323-4ae79d19a5ee","Type":"ContainerDied","Data":"84a8fa74aa8356fd78e53fb22655bbbccf8f4402703e638b4faea92d1bf8c820"} Oct 31 02:05:20.436626 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.436597 5756 scope.go:122] "RemoveContainer" containerID="e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86" Oct 31 02:05:20.437277 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:20.437249 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86\": container with ID starting with e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86 not found: ID does not exist" containerID="e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86" Oct 31 02:05:20.437361 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.437288 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86"} err="failed to get container status \"e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86\": rpc error: code = NotFound desc = could not find container \"e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86\": container with ID starting with e0204dd0c15777d9bcd99bdbc544fbc21fbf03c679ffa1e451d94d2c92d2db86 not found: ID does not exist" Oct 31 02:05:20.494256 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.494225 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume\") pod \"6aa99ba2-4df3-41b0-92d0-92d445980db2\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " Oct 31 02:05:20.494429 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.494360 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb\" (UniqueName: \"kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb\") pod \"6aa99ba2-4df3-41b0-92d0-92d445980db2\" (UID: \"6aa99ba2-4df3-41b0-92d0-92d445980db2\") " Oct 31 02:05:20.495296 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.495255 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume" pod "6aa99ba2-4df3-41b0-92d0-92d445980db2" (UID: "6aa99ba2-4df3-41b0-92d0-92d445980db2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:20.497250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.497229 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb" pod "6aa99ba2-4df3-41b0-92d0-92d445980db2" (UID: "6aa99ba2-4df3-41b0-92d0-92d445980db2"). InnerVolumeSpecName "kube-api-access-tsqbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:20.507785 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.507762 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 02:05:20.595506 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.595469 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr\" (UniqueName: \"kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr\") pod \"006578cd-b9d3-405e-a323-4ae79d19a5ee\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " Oct 31 02:05:20.595706 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.595522 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume\") pod \"006578cd-b9d3-405e-a323-4ae79d19a5ee\" (UID: \"006578cd-b9d3-405e-a323-4ae79d19a5ee\") " Oct 31 02:05:20.596023 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.595994 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume" pod "006578cd-b9d3-405e-a323-4ae79d19a5ee" (UID: "006578cd-b9d3-405e-a323-4ae79d19a5ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:20.596306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.596212 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/006578cd-b9d3-405e-a323-4ae79d19a5ee-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:20.596306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.596231 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aa99ba2-4df3-41b0-92d0-92d445980db2-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:20.596306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.596241 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqbb\" (UniqueName: \"kubernetes.io/projected/6aa99ba2-4df3-41b0-92d0-92d445980db2-kube-api-access-tsqbb\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:20.598058 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.598034 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr" pod "006578cd-b9d3-405e-a323-4ae79d19a5ee" (UID: "006578cd-b9d3-405e-a323-4ae79d19a5ee"). InnerVolumeSpecName "kube-api-access-kjnkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:20.697540 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.697460 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjnkr\" (UniqueName: \"kubernetes.io/projected/006578cd-b9d3-405e-a323-4ae79d19a5ee-kube-api-access-kjnkr\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:20.848026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.847944 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw"] Oct 31 02:05:20.851761 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:20.851733 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6c6990-6666c66857-9z5gw"] Oct 31 02:05:21.434685 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:21.434597 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" event={"ID":"006578cd-b9d3-405e-a323-4ae79d19a5ee","Type":"ContainerDied","Data":"050e9e851a8bc99ee8f079b6d6d5bfa5bbcac429dc1f70b89d810ae16ae1cb3d"} Oct 31 02:05:21.434685 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:21.434617 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr" Oct 31 02:05:21.434685 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:21.434664 5756 scope.go:122] "RemoveContainer" containerID="84a8fa74aa8356fd78e53fb22655bbbccf8f4402703e638b4faea92d1bf8c820" Oct 31 02:05:21.597188 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:21.597153 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr"] Oct 31 02:05:21.599879 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:21.599833 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6c6990-57fcf755c5-rfthr"] Oct 31 02:05:22.321010 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:22.320973 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006578cd-b9d3-405e-a323-4ae79d19a5ee" path="/var/lib/kubelet/pods/006578cd-b9d3-405e-a323-4ae79d19a5ee/volumes" Oct 31 02:05:22.321421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:22.321406 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa99ba2-4df3-41b0-92d0-92d445980db2" path="/var/lib/kubelet/pods/6aa99ba2-4df3-41b0-92d0-92d445980db2/volumes" Oct 31 02:05:22.704416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:22.704330 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc"] Oct 31 02:05:22.753393 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:22.753359 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d50516-h5qlc"] Oct 31 02:05:24.312485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:24.312451 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f498b8-91fe-4142-88e9-4d7456162fc8" path="/var/lib/kubelet/pods/57f498b8-91fe-4142-88e9-4d7456162fc8/volumes" Oct 31 02:05:27.423047 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.422997 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs"] Oct 31 02:05:27.424162 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424137 5756 cpu_manager.go:414] "containerMap: removing container" podUID="006578cd-b9d3-405e-a323-4ae79d19a5ee" containerName="payload-pre-processing" Oct 31 02:05:27.424162 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424162 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006578cd-b9d3-405e-a323-4ae79d19a5ee" containerName="payload-pre-processing" Oct 31 02:05:27.424378 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424176 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bcb66553-b7b7-43b3-b3de-32f108db5558" containerName="revoke-keys" Oct 31 02:05:27.424378 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424185 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb66553-b7b7-43b3-b3de-32f108db5558" containerName="revoke-keys" Oct 31 02:05:27.424378 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424211 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6aa99ba2-4df3-41b0-92d0-92d445980db2" containerName="payload-processing" Oct 31 02:05:27.424378 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424218 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa99ba2-4df3-41b0-92d0-92d445980db2" containerName="payload-processing" Oct 31 02:05:27.424623 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424597 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="006578cd-b9d3-405e-a323-4ae79d19a5ee" containerName="payload-pre-processing" Oct 31 02:05:27.424623 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424618 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb66553-b7b7-43b3-b3de-32f108db5558" containerName="revoke-keys" Oct 31 02:05:27.424831 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.424630 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa99ba2-4df3-41b0-92d0-92d445980db2" containerName="payload-processing" Oct 31 02:05:27.428601 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.428578 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.483510 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.483481 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs"] Oct 31 02:05:27.529225 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.529192 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:05:27.529396 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.529293 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:05:27.587033 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.587003 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.587033 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.587036 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.587268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.587148 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm95d\" (UniqueName: \"kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.688657 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.688571 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.688657 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.688608 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.688949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.688664 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm95d\" (UniqueName: \"kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.689346 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.689326 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.691115 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.691095 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:27.745758 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:27.745722 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm95d\" (UniqueName: \"kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d\") pod \"maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:28.041619 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:28.041591 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:28.433295 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:28.433270 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs"] Oct 31 02:05:28.434756 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:05:28.434728 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3f545e_bed1_4476_a24b_7bedf4b8c0e1.slice/crio-004abfd94d5c3511896e520a828680482aae3e800ff0fac85cc0357f28b2b225 WatchSource:0}: Error finding container 004abfd94d5c3511896e520a828680482aae3e800ff0fac85cc0357f28b2b225: Status 404 returned error can't find the container with id 004abfd94d5c3511896e520a828680482aae3e800ff0fac85cc0357f28b2b225 Oct 31 02:05:28.491500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:28.491467 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" event={"ID":"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1","Type":"ContainerStarted","Data":"004abfd94d5c3511896e520a828680482aae3e800ff0fac85cc0357f28b2b225"} Oct 31 02:05:29.310034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:29.310002 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw"] Oct 31 02:05:29.310222 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:29.310201 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" containerID="cri-o://15728789ed4815da014b7ccabd709ead00d9c57b062f7455b672299ac29c6d7b" gracePeriod=30 Oct 31 02:05:29.500016 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:29.499982 5756 generic.go:356] "Generic (PLEG): container finished" podID="8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" containerID="e9be40d357151bc377ea52af7d7a964a7c21a56e709acdea59296ba18a1a7c35" exitCode=0 Oct 31 02:05:29.500401 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:29.500065 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" event={"ID":"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1","Type":"ContainerDied","Data":"e9be40d357151bc377ea52af7d7a964a7c21a56e709acdea59296ba18a1a7c35"} Oct 31 02:05:30.659383 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.659352 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:30.831479 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.831445 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca\") pod \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " Oct 31 02:05:30.831692 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.831527 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d\" (UniqueName: \"kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d\") pod \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " Oct 31 02:05:30.831755 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.831729 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca\") pod \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\" (UID: \"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1\") " Oct 31 02:05:30.831904 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.831818 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca" pod "8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" (UID: "8c3f545e-bed1-4476-a24b-7bedf4b8c0e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:30.832255 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.832238 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:30.833946 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.833925 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca" pod "8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" (UID: "8c3f545e-bed1-4476-a24b-7bedf4b8c0e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:05:30.834026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.833973 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d" pod "8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" (UID: "8c3f545e-bed1-4476-a24b-7bedf4b8c0e1"). InnerVolumeSpecName "kube-api-access-gm95d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:30.933373 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.933331 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm95d\" (UniqueName: \"kubernetes.io/projected/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-kube-api-access-gm95d\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:30.933373 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:30.933367 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:31.078751 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.078716 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm"] Oct 31 02:05:31.079411 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.079396 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" containerName="revoke-keys" Oct 31 02:05:31.079460 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.079417 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" containerName="revoke-keys" Oct 31 02:05:31.079585 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.079575 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" containerName="revoke-keys" Oct 31 02:05:31.081966 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.081921 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.139706 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.139671 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm"] Oct 31 02:05:31.237912 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.237842 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.238083 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.238059 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jmn\" (UniqueName: \"kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.238173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.238112 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.339689 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.339604 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.339885 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.339709 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jmn\" (UniqueName: \"kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.339885 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.339741 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.340475 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.340409 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.342246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.342220 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.395891 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.395847 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jmn\" (UniqueName: \"kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn\") pod \"maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.406769 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.406744 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:31.515520 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.515487 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" Oct 31 02:05:31.515520 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.515509 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs" event={"ID":"8c3f545e-bed1-4476-a24b-7bedf4b8c0e1","Type":"ContainerDied","Data":"004abfd94d5c3511896e520a828680482aae3e800ff0fac85cc0357f28b2b225"} Oct 31 02:05:31.515719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.515562 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="004abfd94d5c3511896e520a828680482aae3e800ff0fac85cc0357f28b2b225" Oct 31 02:05:31.795158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:31.795114 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm"] Oct 31 02:05:32.068367 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:32.068330 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:05:32.523686 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:32.523590 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" event={"ID":"260d9a2e-59b3-4468-bce8-3a8c69f817d6","Type":"ContainerStarted","Data":"76fe6cf284e2d6e7ef3c6da00927816daae95f7f2dfdb717ee44a73b6dbc15a9"} Oct 31 02:05:32.523686 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:32.523654 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" event={"ID":"260d9a2e-59b3-4468-bce8-3a8c69f817d6","Type":"ContainerStarted","Data":"be37543e7dea2be237e9077219a999c744a2b34a072bedea1666afb81189460e"} Oct 31 02:05:32.639110 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:32.639061 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" podStartSLOduration=1.639046897 podStartE2EDuration="1.639046897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:32.63596481 +0000 UTC m=+4104.442932346" watchObservedRunningTime="2026-10-31 02:05:32.639046897 +0000 UTC m=+4104.446014440" Oct 31 02:05:33.531984 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:33.531948 5756 generic.go:356] "Generic (PLEG): container finished" podID="260d9a2e-59b3-4468-bce8-3a8c69f817d6" containerID="76fe6cf284e2d6e7ef3c6da00927816daae95f7f2dfdb717ee44a73b6dbc15a9" exitCode=0 Oct 31 02:05:33.532400 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:33.532036 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" event={"ID":"260d9a2e-59b3-4468-bce8-3a8c69f817d6","Type":"ContainerDied","Data":"76fe6cf284e2d6e7ef3c6da00927816daae95f7f2dfdb717ee44a73b6dbc15a9"} Oct 31 02:05:34.544105 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.544074 5756 generic.go:356] "Generic (PLEG): container finished" podID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerID="15728789ed4815da014b7ccabd709ead00d9c57b062f7455b672299ac29c6d7b" exitCode=0 Oct 31 02:05:34.544633 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.544151 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" event={"ID":"2decf12d-18bb-4c12-ad66-dddeb0a47b8e","Type":"ContainerDied","Data":"15728789ed4815da014b7ccabd709ead00d9c57b062f7455b672299ac29c6d7b"} Oct 31 02:05:34.544633 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.544222 5756 scope.go:122] "RemoveContainer" containerID="c211332e90e06a26518ac5799aaa4dfc1cadd4a3ed8c77524aa655eb166b74dd" Oct 31 02:05:34.591023 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.590962 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:05:34.680059 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680033 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680071 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680130 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680172 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680201 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680245 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680353 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:34.680441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680371 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680626 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680443 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680626 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680482 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr\") pod \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\" (UID: \"2decf12d-18bb-4c12-ad66-dddeb0a47b8e\") " Oct 31 02:05:34.680733 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680643 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:34.680897 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680822 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:34.681024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.680992 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:34.681152 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.681043 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.681152 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.681065 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.681152 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.681080 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.681333 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.681251 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:34.683091 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.683072 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:34.683306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.683287 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:05:34.683502 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.683475 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:34.683590 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.683569 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr" pod "2decf12d-18bb-4c12-ad66-dddeb0a47b8e" (UID: "2decf12d-18bb-4c12-ad66-dddeb0a47b8e"). InnerVolumeSpecName "kube-api-access-dlbnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:34.689719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.689701 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:34.782601 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.782563 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca\") pod \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " Oct 31 02:05:34.782804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.782627 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca\") pod \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " Oct 31 02:05:34.782804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.782694 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn\" (UniqueName: \"kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn\") pod \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\" (UID: \"260d9a2e-59b3-4468-bce8-3a8c69f817d6\") " Oct 31 02:05:34.783086 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783053 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca" pod "260d9a2e-59b3-4468-bce8-3a8c69f817d6" (UID: "260d9a2e-59b3-4468-bce8-3a8c69f817d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:34.783473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783416 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbnr\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-kube-api-access-dlbnr\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.783473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783449 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.783473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783464 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.783676 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783478 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.783676 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783492 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.783676 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783505 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.783676 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.783518 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2decf12d-18bb-4c12-ad66-dddeb0a47b8e-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.785498 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.785458 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca" pod "260d9a2e-59b3-4468-bce8-3a8c69f817d6" (UID: "260d9a2e-59b3-4468-bce8-3a8c69f817d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:05:34.785640 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.785618 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn" pod "260d9a2e-59b3-4468-bce8-3a8c69f817d6" (UID: "260d9a2e-59b3-4468-bce8-3a8c69f817d6"). InnerVolumeSpecName "kube-api-access-j8jmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:34.884719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.884637 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260d9a2e-59b3-4468-bce8-3a8c69f817d6-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:34.884719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:34.884663 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8jmn\" (UniqueName: \"kubernetes.io/projected/260d9a2e-59b3-4468-bce8-3a8c69f817d6-kube-api-access-j8jmn\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:35.555300 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.555265 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" Oct 31 02:05:35.555741 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.555298 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm" event={"ID":"260d9a2e-59b3-4468-bce8-3a8c69f817d6","Type":"ContainerDied","Data":"be37543e7dea2be237e9077219a999c744a2b34a072bedea1666afb81189460e"} Oct 31 02:05:35.555741 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.555359 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be37543e7dea2be237e9077219a999c744a2b34a072bedea1666afb81189460e" Oct 31 02:05:35.556879 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.556837 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" event={"ID":"2decf12d-18bb-4c12-ad66-dddeb0a47b8e","Type":"ContainerDied","Data":"7b89102584ee22f3d52fc5262d3e135f0b8e94701cc22597cf81e5693392d867"} Oct 31 02:05:35.557013 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.556912 5756 scope.go:122] "RemoveContainer" containerID="15728789ed4815da014b7ccabd709ead00d9c57b062f7455b672299ac29c6d7b" Oct 31 02:05:35.557013 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.556889 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw" Oct 31 02:05:35.771857 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.771776 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw"] Oct 31 02:05:35.775842 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.775816 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d50516-openshift-default-db6dbb8c7-6k5kw"] Oct 31 02:05:35.781664 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:35.781637 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:05:36.312764 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:36.312733 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" path="/var/lib/kubelet/pods/2decf12d-18bb-4c12-ad66-dddeb0a47b8e/volumes" Oct 31 02:05:38.902926 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:38.902856 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd"] Oct 31 02:05:39.008763 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:39.008309 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr"] Oct 31 02:05:41.979619 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.979583 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22"] Oct 31 02:05:41.980464 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980444 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" Oct 31 02:05:41.980572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980471 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" Oct 31 02:05:41.980572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980499 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" Oct 31 02:05:41.980572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980505 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" Oct 31 02:05:41.980572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980530 5756 cpu_manager.go:414] "containerMap: removing container" podUID="260d9a2e-59b3-4468-bce8-3a8c69f817d6" containerName="revoke-keys" Oct 31 02:05:41.980572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980535 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260d9a2e-59b3-4468-bce8-3a8c69f817d6" containerName="revoke-keys" Oct 31 02:05:41.980846 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980618 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" Oct 31 02:05:41.980846 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980625 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2decf12d-18bb-4c12-ad66-dddeb0a47b8e" containerName="istio-proxy" Oct 31 02:05:41.980846 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.980637 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="260d9a2e-59b3-4468-bce8-3a8c69f817d6" containerName="revoke-keys" Oct 31 02:05:41.983735 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:41.983714 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.036426 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.036390 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a713e6-openshift-default-dockercfg-rr89b\"" Oct 31 02:05:42.071387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071352 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071555 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071401 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071555 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071502 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrv4\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071555 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071550 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071666 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071633 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071704 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071692 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071724 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071760 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.071832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.071803 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.089173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.089140 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22"] Oct 31 02:05:42.173297 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173256 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.173599 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173579 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.173737 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173714 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.173914 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173795 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.173914 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173840 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174087 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173941 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrv4\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174087 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.174010 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174215 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.174096 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174215 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.174145 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174558 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.174529 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174558 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.173640 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.174831 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.174807 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.175169 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.175144 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.175418 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.175394 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.196741 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.196709 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.204922 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.202078 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.230997 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.230923 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.231794 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.231763 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrv4\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4\") pod \"e2e-extmod-a-a713e6-openshift-default-686545875c-59f22\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.298064 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.298024 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:42.649288 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.649253 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997"] Oct 31 02:05:42.755020 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.754979 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk"] Oct 31 02:05:42.784957 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.784912 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22"] Oct 31 02:05:42.815663 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:05:42.815632 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106248d9_94d9_4c0e_8dba_07fe9718d991.slice/crio-a01fef8d228dfd791972b989e0f60b9dc3bf594ac5c651f6a97a03a93d52d361 WatchSource:0}: Error finding container a01fef8d228dfd791972b989e0f60b9dc3bf594ac5c651f6a97a03a93d52d361: Status 404 returned error can't find the container with id a01fef8d228dfd791972b989e0f60b9dc3bf594ac5c651f6a97a03a93d52d361 Oct 31 02:05:42.818724 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.818688 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:05:42.818809 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.818780 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:05:42.818852 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:42.818826 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:05:43.613355 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:43.613314 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" event={"ID":"106248d9-94d9-4c0e-8dba-07fe9718d991","Type":"ContainerStarted","Data":"8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d"} Oct 31 02:05:43.613899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:43.613381 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" event={"ID":"106248d9-94d9-4c0e-8dba-07fe9718d991","Type":"ContainerStarted","Data":"a01fef8d228dfd791972b989e0f60b9dc3bf594ac5c651f6a97a03a93d52d361"} Oct 31 02:05:43.731668 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:43.731603 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podStartSLOduration=2.731585158 podStartE2EDuration="2.731585158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:43.729972107 +0000 UTC m=+4115.536939650" watchObservedRunningTime="2026-10-31 02:05:43.731585158 +0000 UTC m=+4115.538552703" Oct 31 02:05:43.903620 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:43.903543 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" podUID="6ac65895-9240-4575-b70f-91ce321af50a" containerName="payload-processing" containerID="cri-o://6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93" gracePeriod=25 Oct 31 02:05:44.009383 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.009335 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" podUID="98e28427-6684-4ad9-8978-5e6438db9548" containerName="payload-pre-processing" containerID="cri-o://650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205" gracePeriod=25 Oct 31 02:05:44.226291 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.226265 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 02:05:44.305896 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.299058 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:44.305896 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.301714 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc\" (UniqueName: \"kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc\") pod \"6ac65895-9240-4575-b70f-91ce321af50a\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " Oct 31 02:05:44.305896 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.302074 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume\") pod \"6ac65895-9240-4575-b70f-91ce321af50a\" (UID: \"6ac65895-9240-4575-b70f-91ce321af50a\") " Oct 31 02:05:44.305896 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.303542 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume" pod "6ac65895-9240-4575-b70f-91ce321af50a" (UID: "6ac65895-9240-4575-b70f-91ce321af50a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:44.313829 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.313778 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc" pod "6ac65895-9240-4575-b70f-91ce321af50a" (UID: "6ac65895-9240-4575-b70f-91ce321af50a"). InnerVolumeSpecName "kube-api-access-889tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:44.317781 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.317737 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:44.345753 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.345212 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 02:05:44.403458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.403417 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume\") pod \"98e28427-6684-4ad9-8978-5e6438db9548\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " Oct 31 02:05:44.403675 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.403603 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt\" (UniqueName: \"kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt\") pod \"98e28427-6684-4ad9-8978-5e6438db9548\" (UID: \"98e28427-6684-4ad9-8978-5e6438db9548\") " Oct 31 02:05:44.404180 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.404099 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac65895-9240-4575-b70f-91ce321af50a-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:44.404784 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.404123 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-889tc\" (UniqueName: \"kubernetes.io/projected/6ac65895-9240-4575-b70f-91ce321af50a-kube-api-access-889tc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:44.405600 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.405574 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume" pod "98e28427-6684-4ad9-8978-5e6438db9548" (UID: "98e28427-6684-4ad9-8978-5e6438db9548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:44.415253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.413724 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt" pod "98e28427-6684-4ad9-8978-5e6438db9548" (UID: "98e28427-6684-4ad9-8978-5e6438db9548"). InnerVolumeSpecName "kube-api-access-bxtkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:44.505601 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.505561 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxtkt\" (UniqueName: \"kubernetes.io/projected/98e28427-6684-4ad9-8978-5e6438db9548-kube-api-access-bxtkt\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:44.505601 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.505595 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98e28427-6684-4ad9-8978-5e6438db9548-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:44.620752 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.620714 5756 generic.go:356] "Generic (PLEG): container finished" podID="98e28427-6684-4ad9-8978-5e6438db9548" containerID="650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205" exitCode=0 Oct 31 02:05:44.621210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.620772 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" Oct 31 02:05:44.621210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.620807 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" event={"ID":"98e28427-6684-4ad9-8978-5e6438db9548","Type":"ContainerDied","Data":"650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205"} Oct 31 02:05:44.621210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.620904 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr" event={"ID":"98e28427-6684-4ad9-8978-5e6438db9548","Type":"ContainerDied","Data":"d5995f3071044aa90bfe66faa6ea1d597f561d39db52c2e9d1bc72741503e0f2"} Oct 31 02:05:44.621210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.620940 5756 scope.go:122] "RemoveContainer" containerID="650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205" Oct 31 02:05:44.622285 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.622266 5756 generic.go:356] "Generic (PLEG): container finished" podID="6ac65895-9240-4575-b70f-91ce321af50a" containerID="6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93" exitCode=0 Oct 31 02:05:44.622405 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.622301 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" event={"ID":"6ac65895-9240-4575-b70f-91ce321af50a","Type":"ContainerDied","Data":"6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93"} Oct 31 02:05:44.622405 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.622334 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" Oct 31 02:05:44.622405 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.622347 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd" event={"ID":"6ac65895-9240-4575-b70f-91ce321af50a","Type":"ContainerDied","Data":"ed0525f543ed3b54a83759abfd7e9dc0ead87e13cdd011a0144c9a0a48c49afb"} Oct 31 02:05:44.622825 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.622799 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:44.623812 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.623792 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:05:44.635547 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.635522 5756 scope.go:122] "RemoveContainer" containerID="650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205" Oct 31 02:05:44.635812 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:44.635793 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205\": container with ID starting with 650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205 not found: ID does not exist" containerID="650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205" Oct 31 02:05:44.635899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.635820 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205"} err="failed to get container status \"650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205\": rpc error: code = NotFound desc = could not find container \"650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205\": container with ID starting with 650e4a86558fe4604f068df6602c51d5ab77dde480fff08e77ca3d79357a9205 not found: ID does not exist" Oct 31 02:05:44.635899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.635837 5756 scope.go:122] "RemoveContainer" containerID="6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93" Oct 31 02:05:44.645690 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.645674 5756 scope.go:122] "RemoveContainer" containerID="6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93" Oct 31 02:05:44.646002 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:44.645985 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93\": container with ID starting with 6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93 not found: ID does not exist" containerID="6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93" Oct 31 02:05:44.646063 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.646008 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93"} err="failed to get container status \"6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93\": rpc error: code = NotFound desc = could not find container \"6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93\": container with ID starting with 6d10efbc22cff32f7d00e0e2ed3c60a0bafefd1c4ef4814848e9ceea59763b93 not found: ID does not exist" Oct 31 02:05:44.784488 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.784462 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr"] Oct 31 02:05:44.790516 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:44.790491 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6caf19-5cf4b5dfd7-lrhfr"] Oct 31 02:05:45.017546 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:45.017510 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd"] Oct 31 02:05:45.021198 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:45.021172 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6caf19-7567445fcf-5x4fd"] Oct 31 02:05:46.312006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:46.311975 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac65895-9240-4575-b70f-91ce321af50a" path="/var/lib/kubelet/pods/6ac65895-9240-4575-b70f-91ce321af50a/volumes" Oct 31 02:05:46.312417 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:46.312403 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e28427-6684-4ad9-8978-5e6438db9548" path="/var/lib/kubelet/pods/98e28427-6684-4ad9-8978-5e6438db9548/volumes" Oct 31 02:05:47.650239 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:47.650196 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" podUID="81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" containerName="payload-processing" containerID="cri-o://b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b" gracePeriod=25 Oct 31 02:05:47.756039 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:47.755978 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" podUID="27677ad2-39d3-4fa5-9fc5-bd1d602e1401" containerName="payload-pre-processing" containerID="cri-o://e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66" gracePeriod=25 Oct 31 02:05:47.995105 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:47.995080 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 02:05:48.039161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.039133 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 02:05:48.051825 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.051793 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h\" (UniqueName: \"kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h\") pod \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " Oct 31 02:05:48.052085 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.052066 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume\") pod \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\" (UID: \"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b\") " Oct 31 02:05:48.052418 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.052394 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume" pod "81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" (UID: "81d43d6f-7e13-4ab6-a1f7-faf4f219d72b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:48.052661 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.052642 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.054043 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.054008 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h" pod "81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" (UID: "81d43d6f-7e13-4ab6-a1f7-faf4f219d72b"). InnerVolumeSpecName "kube-api-access-gzq5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.153660 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.153631 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume\") pod \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " Oct 31 02:05:48.153845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.153831 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752\" (UniqueName: \"kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752\") pod \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\" (UID: \"27677ad2-39d3-4fa5-9fc5-bd1d602e1401\") " Oct 31 02:05:48.154092 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.154068 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume" pod "27677ad2-39d3-4fa5-9fc5-bd1d602e1401" (UID: "27677ad2-39d3-4fa5-9fc5-bd1d602e1401"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:48.154370 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.154349 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.154439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.154377 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzq5h\" (UniqueName: \"kubernetes.io/projected/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b-kube-api-access-gzq5h\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.156189 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.156167 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752" pod "27677ad2-39d3-4fa5-9fc5-bd1d602e1401" (UID: "27677ad2-39d3-4fa5-9fc5-bd1d602e1401"). InnerVolumeSpecName "kube-api-access-df752". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.256004 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.255929 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df752\" (UniqueName: \"kubernetes.io/projected/27677ad2-39d3-4fa5-9fc5-bd1d602e1401-kube-api-access-df752\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.656308 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.656272 5756 generic.go:356] "Generic (PLEG): container finished" podID="81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" containerID="b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b" exitCode=0 Oct 31 02:05:48.656807 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.656326 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" Oct 31 02:05:48.656807 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.656373 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" event={"ID":"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b","Type":"ContainerDied","Data":"b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b"} Oct 31 02:05:48.656807 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.656430 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997" event={"ID":"81d43d6f-7e13-4ab6-a1f7-faf4f219d72b","Type":"ContainerDied","Data":"dbff5bd5769309c889895ad8e969d36e6893f4988cdf6e01ad879a7b34c32cf4"} Oct 31 02:05:48.656807 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.656458 5756 scope.go:122] "RemoveContainer" containerID="b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b" Oct 31 02:05:48.658045 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.658025 5756 generic.go:356] "Generic (PLEG): container finished" podID="27677ad2-39d3-4fa5-9fc5-bd1d602e1401" containerID="e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66" exitCode=0 Oct 31 02:05:48.658142 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.658067 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" Oct 31 02:05:48.658202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.658137 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" event={"ID":"27677ad2-39d3-4fa5-9fc5-bd1d602e1401","Type":"ContainerDied","Data":"e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66"} Oct 31 02:05:48.658202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.658195 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk" event={"ID":"27677ad2-39d3-4fa5-9fc5-bd1d602e1401","Type":"ContainerDied","Data":"39cda8132aa6a4ba06714213cb63982711120cff58f37fdfe45fd5c53729c29c"} Oct 31 02:05:48.675889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.675843 5756 scope.go:122] "RemoveContainer" containerID="b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b" Oct 31 02:05:48.676173 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:48.676154 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b\": container with ID starting with b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b not found: ID does not exist" containerID="b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b" Oct 31 02:05:48.676261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.676186 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b"} err="failed to get container status \"b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b\": rpc error: code = NotFound desc = could not find container \"b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b\": container with ID starting with b22fbd249ec03fee3c17c065d30814a4a7c4b6dedebb9d34dad5f767ddf5a43b not found: ID does not exist" Oct 31 02:05:48.676261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.676211 5756 scope.go:122] "RemoveContainer" containerID="e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66" Oct 31 02:05:48.685701 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.685680 5756 scope.go:122] "RemoveContainer" containerID="e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66" Oct 31 02:05:48.686032 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:05:48.686004 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66\": container with ID starting with e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66 not found: ID does not exist" containerID="e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66" Oct 31 02:05:48.686110 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.686044 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66"} err="failed to get container status \"e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66\": rpc error: code = NotFound desc = could not find container \"e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66\": container with ID starting with e2bf523535527e4efe7ab323156cd0cac42f702a764252033fdf7f41efaf8a66 not found: ID does not exist" Oct 31 02:05:48.820422 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.820387 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk"] Oct 31 02:05:48.823202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.823175 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce8dbb-7445847b47-6rsdk"] Oct 31 02:05:48.936051 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.934240 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997"] Oct 31 02:05:48.940026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.939992 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz"] Oct 31 02:05:48.943203 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.943184 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce8dbb-64d8f4b55f-z4997"] Oct 31 02:05:48.945403 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:48.945381 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-864622-fxqrz"] Oct 31 02:05:49.044949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:49.044916 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv"] Oct 31 02:05:49.047444 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:49.047419 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a9010d8-lc9gv"] Oct 31 02:05:50.311940 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:50.311894 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27677ad2-39d3-4fa5-9fc5-bd1d602e1401" path="/var/lib/kubelet/pods/27677ad2-39d3-4fa5-9fc5-bd1d602e1401/volumes" Oct 31 02:05:50.312559 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:50.312492 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" path="/var/lib/kubelet/pods/81d43d6f-7e13-4ab6-a1f7-faf4f219d72b/volumes" Oct 31 02:05:50.312977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:50.312956 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884690f1-3600-479f-86eb-918de71ef9d5" path="/var/lib/kubelet/pods/884690f1-3600-479f-86eb-918de71ef9d5/volumes" Oct 31 02:05:50.313686 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:50.313665 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9220da3-2136-43c4-be84-3b0448ad0dd9" path="/var/lib/kubelet/pods/c9220da3-2136-43c4-be84-3b0448ad0dd9/volumes" Oct 31 02:05:58.161594 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.161562 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq"] Oct 31 02:05:58.162173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162157 5756 cpu_manager.go:414] "containerMap: removing container" podUID="27677ad2-39d3-4fa5-9fc5-bd1d602e1401" containerName="payload-pre-processing" Oct 31 02:05:58.162220 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162175 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27677ad2-39d3-4fa5-9fc5-bd1d602e1401" containerName="payload-pre-processing" Oct 31 02:05:58.162220 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162213 5756 cpu_manager.go:414] "containerMap: removing container" podUID="81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" containerName="payload-processing" Oct 31 02:05:58.162220 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162218 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" containerName="payload-processing" Oct 31 02:05:58.162317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162228 5756 cpu_manager.go:414] "containerMap: removing container" podUID="98e28427-6684-4ad9-8978-5e6438db9548" containerName="payload-pre-processing" Oct 31 02:05:58.162317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162233 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e28427-6684-4ad9-8978-5e6438db9548" containerName="payload-pre-processing" Oct 31 02:05:58.162317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162244 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6ac65895-9240-4575-b70f-91ce321af50a" containerName="payload-processing" Oct 31 02:05:58.162317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162250 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac65895-9240-4575-b70f-91ce321af50a" containerName="payload-processing" Oct 31 02:05:58.162441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162350 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d43d6f-7e13-4ab6-a1f7-faf4f219d72b" containerName="payload-processing" Oct 31 02:05:58.162441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162360 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e28427-6684-4ad9-8978-5e6438db9548" containerName="payload-pre-processing" Oct 31 02:05:58.162441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162367 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac65895-9240-4575-b70f-91ce321af50a" containerName="payload-processing" Oct 31 02:05:58.162441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.162378 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="27677ad2-39d3-4fa5-9fc5-bd1d602e1401" containerName="payload-pre-processing" Oct 31 02:05:58.167186 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.167169 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.185000 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.184966 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.185174 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.185090 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgwm\" (UniqueName: \"kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.185246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.185169 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.230919 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.226533 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq"] Oct 31 02:05:58.272506 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.272478 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:05:58.272915 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.272851 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:05:58.286037 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.286005 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.286192 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.286075 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgwm\" (UniqueName: \"kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.286192 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.286140 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.286649 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.286627 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.288671 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.288643 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.325476 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.325439 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f"] Oct 31 02:05:58.329756 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.329730 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.343298 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.343270 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgwm\" (UniqueName: \"kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm\") pod \"maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.357535 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.357510 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f"] Oct 31 02:05:58.384589 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.384561 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a713e6-dockercfg-hffn9\"" Oct 31 02:05:58.384792 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.384645 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a713e6\"" Oct 31 02:05:58.387388 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.387356 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.387556 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.387512 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktb7z\" (UniqueName: \"kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z\") pod \"payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.463454 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.463372 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf"] Oct 31 02:05:58.467770 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.467748 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.486530 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.486506 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:05:58.489128 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.489095 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktb7z\" (UniqueName: \"kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z\") pod \"payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.489260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.489248 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.489990 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.489972 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.525289 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.525257 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf"] Oct 31 02:05:58.547694 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.547245 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktb7z\" (UniqueName: \"kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z\") pod \"payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.590569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.590532 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.590759 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.590693 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj2zw\" (UniqueName: \"kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw\") pod \"payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.644595 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.644568 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:58.691912 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.691847 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj2zw\" (UniqueName: \"kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw\") pod \"payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.692094 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.692024 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.692695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.692675 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.748387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.748322 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj2zw\" (UniqueName: \"kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw\") pod \"payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.782081 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.782048 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:58.871316 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:58.871284 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq"] Oct 31 02:05:58.872845 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:05:58.872819 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b04d2a_51bb_4ff7_a1aa_32653722eef4.slice/crio-76502026b1dc89211448916a2f7b4e30fa2a1cdb71c82471d5f8cb6c92f155ea WatchSource:0}: Error finding container 76502026b1dc89211448916a2f7b4e30fa2a1cdb71c82471d5f8cb6c92f155ea: Status 404 returned error can't find the container with id 76502026b1dc89211448916a2f7b4e30fa2a1cdb71c82471d5f8cb6c92f155ea Oct 31 02:05:59.023469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.023438 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f"] Oct 31 02:05:59.024632 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:05:59.024601 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77012e9c_e3d0_4e95_9506_99f2785a793e.slice/crio-e55afa2c00d49a40d51adc998854a3875914bb40438118acdcab85f3e1ad7ec1 WatchSource:0}: Error finding container e55afa2c00d49a40d51adc998854a3875914bb40438118acdcab85f3e1ad7ec1: Status 404 returned error can't find the container with id e55afa2c00d49a40d51adc998854a3875914bb40438118acdcab85f3e1ad7ec1 Oct 31 02:05:59.161425 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.161396 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf"] Oct 31 02:05:59.162873 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:05:59.162835 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99d1ff5_a9fb_428b_bd03_8996f9dd743c.slice/crio-db3ed885f3623e78b7511b96bb5b8bdae4dfd42928a22e7aeadd590ee958c06a WatchSource:0}: Error finding container db3ed885f3623e78b7511b96bb5b8bdae4dfd42928a22e7aeadd590ee958c06a: Status 404 returned error can't find the container with id db3ed885f3623e78b7511b96bb5b8bdae4dfd42928a22e7aeadd590ee958c06a Oct 31 02:05:59.741734 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.741703 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" event={"ID":"d5b04d2a-51bb-4ff7-a1aa-32653722eef4","Type":"ContainerStarted","Data":"9c185a400f29607a1f0621aa5cf483a67ffa44c0ea00e73ff9a806eb494179f0"} Oct 31 02:05:59.741909 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.741765 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" event={"ID":"d5b04d2a-51bb-4ff7-a1aa-32653722eef4","Type":"ContainerStarted","Data":"76502026b1dc89211448916a2f7b4e30fa2a1cdb71c82471d5f8cb6c92f155ea"} Oct 31 02:05:59.743631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.743606 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" event={"ID":"77012e9c-e3d0-4e95-9506-99f2785a793e","Type":"ContainerStarted","Data":"a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9"} Oct 31 02:05:59.743746 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.743648 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" event={"ID":"77012e9c-e3d0-4e95-9506-99f2785a793e","Type":"ContainerStarted","Data":"e55afa2c00d49a40d51adc998854a3875914bb40438118acdcab85f3e1ad7ec1"} Oct 31 02:05:59.743806 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.743760 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:05:59.745294 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.745275 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" event={"ID":"b99d1ff5-a9fb-428b-bd03-8996f9dd743c","Type":"ContainerStarted","Data":"a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817"} Oct 31 02:05:59.745384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.745312 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:05:59.745384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.745329 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" event={"ID":"b99d1ff5-a9fb-428b-bd03-8996f9dd743c","Type":"ContainerStarted","Data":"db3ed885f3623e78b7511b96bb5b8bdae4dfd42928a22e7aeadd590ee958c06a"} Oct 31 02:05:59.989938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:05:59.989801 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" podStartSLOduration=1.989781928 podStartE2EDuration="1.989781928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:59.989295526 +0000 UTC m=+4131.796263071" watchObservedRunningTime="2026-10-31 02:05:59.989781928 +0000 UTC m=+4131.796749473" Oct 31 02:06:00.056782 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.056750 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct"] Oct 31 02:06:00.059557 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.059529 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90277319-kgkct"] Oct 31 02:06:00.104927 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.104876 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" podStartSLOduration=2.104844697 podStartE2EDuration="2.104844697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:06:00.103928787 +0000 UTC m=+4131.910896318" watchObservedRunningTime="2026-10-31 02:06:00.104844697 +0000 UTC m=+4131.911812240" Oct 31 02:06:00.312592 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.312561 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e959bbe6-17bb-4802-b4db-e921dc9d609a" path="/var/lib/kubelet/pods/e959bbe6-17bb-4802-b4db-e921dc9d609a/volumes" Oct 31 02:06:00.753699 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.753600 5756 generic.go:356] "Generic (PLEG): container finished" podID="d5b04d2a-51bb-4ff7-a1aa-32653722eef4" containerID="9c185a400f29607a1f0621aa5cf483a67ffa44c0ea00e73ff9a806eb494179f0" exitCode=0 Oct 31 02:06:00.753699 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.753664 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" event={"ID":"d5b04d2a-51bb-4ff7-a1aa-32653722eef4","Type":"ContainerDied","Data":"9c185a400f29607a1f0621aa5cf483a67ffa44c0ea00e73ff9a806eb494179f0"} Oct 31 02:06:00.910839 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.910810 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:06:00.922584 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.922557 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca\") pod \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " Oct 31 02:06:00.922744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.922711 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm\" (UniqueName: \"kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm\") pod \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " Oct 31 02:06:00.922810 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.922764 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca\") pod \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\" (UID: \"d5b04d2a-51bb-4ff7-a1aa-32653722eef4\") " Oct 31 02:06:00.922949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.922925 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca" pod "d5b04d2a-51bb-4ff7-a1aa-32653722eef4" (UID: "d5b04d2a-51bb-4ff7-a1aa-32653722eef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:06:00.923265 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.923246 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:00.924994 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.924969 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca" pod "d5b04d2a-51bb-4ff7-a1aa-32653722eef4" (UID: "d5b04d2a-51bb-4ff7-a1aa-32653722eef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:06:00.925079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:00.925000 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm" pod "d5b04d2a-51bb-4ff7-a1aa-32653722eef4" (UID: "d5b04d2a-51bb-4ff7-a1aa-32653722eef4"). InnerVolumeSpecName "kube-api-access-btgwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:06:01.024431 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:01.024320 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:01.024431 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:01.024360 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btgwm\" (UniqueName: \"kubernetes.io/projected/d5b04d2a-51bb-4ff7-a1aa-32653722eef4-kube-api-access-btgwm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:01.761814 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:01.761788 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" Oct 31 02:06:01.762259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:01.761820 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq" event={"ID":"d5b04d2a-51bb-4ff7-a1aa-32653722eef4","Type":"ContainerDied","Data":"76502026b1dc89211448916a2f7b4e30fa2a1cdb71c82471d5f8cb6c92f155ea"} Oct 31 02:06:01.762259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:01.761886 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76502026b1dc89211448916a2f7b4e30fa2a1cdb71c82471d5f8cb6c92f155ea" Oct 31 02:06:02.067946 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:06:02.067899 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:06:10.757333 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:10.757298 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:06:10.757735 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:10.757381 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:06:12.053384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:12.053345 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww"] Oct 31 02:06:12.571040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:12.571002 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:06:17.054064 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.053964 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" podUID="7c352096-bf01-44b5-b103-87e995864264" containerName="payload-pre-processing" containerID="cri-o://6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc" gracePeriod=25 Oct 31 02:06:17.331015 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.330984 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:06:17.381202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.381164 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8\" (UniqueName: \"kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8\") pod \"7c352096-bf01-44b5-b103-87e995864264\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " Oct 31 02:06:17.381385 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.381300 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume\") pod \"7c352096-bf01-44b5-b103-87e995864264\" (UID: \"7c352096-bf01-44b5-b103-87e995864264\") " Oct 31 02:06:17.381747 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.381713 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume" pod "7c352096-bf01-44b5-b103-87e995864264" (UID: "7c352096-bf01-44b5-b103-87e995864264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:06:17.381922 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.381902 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c352096-bf01-44b5-b103-87e995864264-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:17.384237 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.384214 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8" pod "7c352096-bf01-44b5-b103-87e995864264" (UID: "7c352096-bf01-44b5-b103-87e995864264"). InnerVolumeSpecName "kube-api-access-g67n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:06:17.482643 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.482607 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67n8\" (UniqueName: \"kubernetes.io/projected/7c352096-bf01-44b5-b103-87e995864264-kube-api-access-g67n8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:17.873537 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.873505 5756 generic.go:356] "Generic (PLEG): container finished" podID="7c352096-bf01-44b5-b103-87e995864264" containerID="6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc" exitCode=0 Oct 31 02:06:17.873700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.873550 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" Oct 31 02:06:17.873700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.873592 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" event={"ID":"7c352096-bf01-44b5-b103-87e995864264","Type":"ContainerDied","Data":"6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc"} Oct 31 02:06:17.873700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.873652 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww" event={"ID":"7c352096-bf01-44b5-b103-87e995864264","Type":"ContainerDied","Data":"3f6a6678c2533bd1c59a600631275e96acab5c16b8105872e01464a5b73b043f"} Oct 31 02:06:17.873700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.873679 5756 scope.go:122] "RemoveContainer" containerID="6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc" Oct 31 02:06:17.884183 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.884162 5756 scope.go:122] "RemoveContainer" containerID="6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc" Oct 31 02:06:17.884452 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:06:17.884432 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc\": container with ID starting with 6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc not found: ID does not exist" containerID="6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc" Oct 31 02:06:17.884523 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:17.884466 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc"} err="failed to get container status \"6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc\": rpc error: code = NotFound desc = could not find container \"6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc\": container with ID starting with 6f28101bcbb1382d170a03d3ce0988130611b7f60b2cd1b9940aaf019e3614bc not found: ID does not exist" Oct 31 02:06:18.039334 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:18.039300 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww"] Oct 31 02:06:18.088451 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:18.088423 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fbe06f-689f958cc9-vwmww"] Oct 31 02:06:18.311954 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:18.311916 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c352096-bf01-44b5-b103-87e995864264" path="/var/lib/kubelet/pods/7c352096-bf01-44b5-b103-87e995864264/volumes" Oct 31 02:06:29.639217 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:29.639181 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq"] Oct 31 02:06:29.688288 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:29.688259 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-fbe06f-khpnq"] Oct 31 02:06:30.312584 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:30.312550 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b04d2a-51bb-4ff7-a1aa-32653722eef4" path="/var/lib/kubelet/pods/d5b04d2a-51bb-4ff7-a1aa-32653722eef4/volumes" Oct 31 02:06:32.067522 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:06:32.067484 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:06:45.230386 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:45.230353 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt"] Oct 31 02:06:45.283139 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:45.283106 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6c6990-bg6kt"] Oct 31 02:06:46.311386 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:46.311354 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb66553-b7b7-43b3-b3de-32f108db5558" path="/var/lib/kubelet/pods/bcb66553-b7b7-43b3-b3de-32f108db5558/volumes" Oct 31 02:06:46.394659 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:46.394623 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4"] Oct 31 02:06:46.394889 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:46.394847 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" containerID="cri-o://002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89" gracePeriod=30 Oct 31 02:06:47.639712 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.639670 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt"] Oct 31 02:06:47.640609 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.640589 5756 cpu_manager.go:414] "containerMap: removing container" podUID="7c352096-bf01-44b5-b103-87e995864264" containerName="payload-pre-processing" Oct 31 02:06:47.640669 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.640612 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c352096-bf01-44b5-b103-87e995864264" containerName="payload-pre-processing" Oct 31 02:06:47.640669 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.640629 5756 cpu_manager.go:414] "containerMap: removing container" podUID="d5b04d2a-51bb-4ff7-a1aa-32653722eef4" containerName="revoke-keys" Oct 31 02:06:47.640669 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.640637 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b04d2a-51bb-4ff7-a1aa-32653722eef4" containerName="revoke-keys" Oct 31 02:06:47.640839 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.640827 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c352096-bf01-44b5-b103-87e995864264" containerName="payload-pre-processing" Oct 31 02:06:47.640901 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.640844 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b04d2a-51bb-4ff7-a1aa-32653722eef4" containerName="revoke-keys" Oct 31 02:06:47.645123 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.645098 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.699054 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.699022 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt"] Oct 31 02:06:47.747033 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.746994 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:06:47.747211 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.747040 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:06:47.761597 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.761521 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.761597 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.761593 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.761839 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.761778 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds6f\" (UniqueName: \"kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.863402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.863371 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.863586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.863485 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds6f\" (UniqueName: \"kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.863645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.863590 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.864247 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.864226 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.866188 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.866166 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.920615 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.920540 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds6f\" (UniqueName: \"kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f\") pod \"maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:47.959106 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:47.959076 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:48.337002 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:06:48.336933 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9234a39f_dc39_49f2_9200_dbdc34a9bf98.slice/crio-15606988eab4233eb7e6b55be65497fef0acf7667829a16828f9d3b4fa282292 WatchSource:0}: Error finding container 15606988eab4233eb7e6b55be65497fef0acf7667829a16828f9d3b4fa282292: Status 404 returned error can't find the container with id 15606988eab4233eb7e6b55be65497fef0acf7667829a16828f9d3b4fa282292 Oct 31 02:06:48.340566 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:48.340542 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt"] Oct 31 02:06:49.109508 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:49.109473 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" event={"ID":"9234a39f-dc39-49f2-9200-dbdc34a9bf98","Type":"ContainerStarted","Data":"88615ef2682f35f8b29b127cbd306086b28e941a26e8016f5d6daa832fd4e6b0"} Oct 31 02:06:49.109910 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:49.109550 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" event={"ID":"9234a39f-dc39-49f2-9200-dbdc34a9bf98","Type":"ContainerStarted","Data":"15606988eab4233eb7e6b55be65497fef0acf7667829a16828f9d3b4fa282292"} Oct 31 02:06:49.221750 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:49.221693 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" podStartSLOduration=2.221680217 podStartE2EDuration="2.221680217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:06:49.221071855 +0000 UTC m=+4181.028039398" watchObservedRunningTime="2026-10-31 02:06:49.221680217 +0000 UTC m=+4181.028647760" Oct 31 02:06:49.650241 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:49.650204 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:06:50.116777 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:50.116742 5756 generic.go:356] "Generic (PLEG): container finished" podID="9234a39f-dc39-49f2-9200-dbdc34a9bf98" containerID="88615ef2682f35f8b29b127cbd306086b28e941a26e8016f5d6daa832fd4e6b0" exitCode=0 Oct 31 02:06:50.117227 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:50.116831 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" event={"ID":"9234a39f-dc39-49f2-9200-dbdc34a9bf98","Type":"ContainerDied","Data":"88615ef2682f35f8b29b127cbd306086b28e941a26e8016f5d6daa832fd4e6b0"} Oct 31 02:06:51.283135 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.283110 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:51.415174 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.415149 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca\") pod \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " Oct 31 02:06:51.415303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.415275 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f\" (UniqueName: \"kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f\") pod \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " Oct 31 02:06:51.415411 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.415395 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca\") pod \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\" (UID: \"9234a39f-dc39-49f2-9200-dbdc34a9bf98\") " Oct 31 02:06:51.415511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.415482 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca" pod "9234a39f-dc39-49f2-9200-dbdc34a9bf98" (UID: "9234a39f-dc39-49f2-9200-dbdc34a9bf98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:06:51.415908 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.415885 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.417730 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.417702 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca" pod "9234a39f-dc39-49f2-9200-dbdc34a9bf98" (UID: "9234a39f-dc39-49f2-9200-dbdc34a9bf98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:06:51.417826 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.417760 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f" pod "9234a39f-dc39-49f2-9200-dbdc34a9bf98" (UID: "9234a39f-dc39-49f2-9200-dbdc34a9bf98"). InnerVolumeSpecName "kube-api-access-tds6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:06:51.518062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.518035 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tds6f\" (UniqueName: \"kubernetes.io/projected/9234a39f-dc39-49f2-9200-dbdc34a9bf98-kube-api-access-tds6f\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.518150 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.518069 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9234a39f-dc39-49f2-9200-dbdc34a9bf98-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.669094 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.669035 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:06:51.681148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.681124 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs"] Oct 31 02:06:51.683613 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.683594 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6caf19-jlnvs"] Oct 31 02:06:51.821597 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821561 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.821790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821633 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.821790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821759 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.821945 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821822 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.821945 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821857 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.821945 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821928 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.822126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.821981 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:06:51.822126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822053 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.822126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822090 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.822298 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822208 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data\") pod \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\" (UID: \"8915ddc2-27bd-4d6e-a6ec-c6d318371863\") " Oct 31 02:06:51.822830 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822626 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:06:51.822830 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822662 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:06:51.822830 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822772 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.823155 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822794 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.823155 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822897 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:06:51.823155 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.822996 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:06:51.824734 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.824708 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:06:51.825157 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.825130 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:06:51.825157 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.825137 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "kube-api-access-875zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:06:51.826449 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.826424 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo" pod "8915ddc2-27bd-4d6e-a6ec-c6d318371863" (UID: "8915ddc2-27bd-4d6e-a6ec-c6d318371863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:06:51.924260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924165 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875zj\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-kube-api-access-875zj\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.924260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924201 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.924260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924216 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.924260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924228 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.924260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924240 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.924260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924254 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.924588 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.924269 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8915ddc2-27bd-4d6e-a6ec-c6d318371863-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:51.944170 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.944140 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q"] Oct 31 02:06:51.949806 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.949776 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92"] Oct 31 02:06:51.962543 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:51.962518 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz"] Oct 31 02:06:52.132832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.132793 5756 generic.go:356] "Generic (PLEG): container finished" podID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerID="002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89" exitCode=0 Oct 31 02:06:52.133027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.132834 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" event={"ID":"8915ddc2-27bd-4d6e-a6ec-c6d318371863","Type":"ContainerDied","Data":"002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89"} Oct 31 02:06:52.133027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.132849 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" Oct 31 02:06:52.133027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.132908 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4" event={"ID":"8915ddc2-27bd-4d6e-a6ec-c6d318371863","Type":"ContainerDied","Data":"003f1404d504827142b723353a689f3f68899eb74333c01e6dc38d0523190746"} Oct 31 02:06:52.133027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.132937 5756 scope.go:122] "RemoveContainer" containerID="002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89" Oct 31 02:06:52.134722 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.134686 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" Oct 31 02:06:52.134722 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.134712 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt" event={"ID":"9234a39f-dc39-49f2-9200-dbdc34a9bf98","Type":"ContainerDied","Data":"15606988eab4233eb7e6b55be65497fef0acf7667829a16828f9d3b4fa282292"} Oct 31 02:06:52.135004 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.134746 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15606988eab4233eb7e6b55be65497fef0acf7667829a16828f9d3b4fa282292" Oct 31 02:06:52.145169 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.145147 5756 scope.go:122] "RemoveContainer" containerID="0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e" Oct 31 02:06:52.155080 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.155062 5756 scope.go:122] "RemoveContainer" containerID="002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89" Oct 31 02:06:52.155385 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:06:52.155366 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89\": container with ID starting with 002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89 not found: ID does not exist" containerID="002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89" Oct 31 02:06:52.155453 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.155393 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89"} err="failed to get container status \"002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89\": rpc error: code = NotFound desc = could not find container \"002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89\": container with ID starting with 002cf4610147d2f98e76fa7be64eeff99935ecee75c77c3b511545433e097d89 not found: ID does not exist" Oct 31 02:06:52.155453 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.155413 5756 scope.go:122] "RemoveContainer" containerID="0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e" Oct 31 02:06:52.155639 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:06:52.155622 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e\": container with ID starting with 0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e not found: ID does not exist" containerID="0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e" Oct 31 02:06:52.155700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.155649 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e"} err="failed to get container status \"0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e\": rpc error: code = NotFound desc = could not find container \"0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e\": container with ID starting with 0ebecce42d54bd35ddc85edebb90ef07f8d15f694caee2851f1078b5b8d28d0e not found: ID does not exist" Oct 31 02:06:52.313083 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.313049 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3f545e-bed1-4476-a24b-7bedf4b8c0e1" path="/var/lib/kubelet/pods/8c3f545e-bed1-4476-a24b-7bedf4b8c0e1/volumes" Oct 31 02:06:52.451945 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.451814 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4"] Oct 31 02:06:52.455837 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:52.455812 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fbe06f-openshift-default-68589cdf-kshv4"] Oct 31 02:06:53.083301 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.083263 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg"] Oct 31 02:06:53.083938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.083914 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" Oct 31 02:06:53.083998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.083941 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" Oct 31 02:06:53.083998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.083958 5756 cpu_manager.go:414] "containerMap: removing container" podUID="9234a39f-dc39-49f2-9200-dbdc34a9bf98" containerName="revoke-keys" Oct 31 02:06:53.083998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.083964 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9234a39f-dc39-49f2-9200-dbdc34a9bf98" containerName="revoke-keys" Oct 31 02:06:53.084136 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.084113 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" Oct 31 02:06:53.084186 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.084149 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" Oct 31 02:06:53.084186 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.084159 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="9234a39f-dc39-49f2-9200-dbdc34a9bf98" containerName="revoke-keys" Oct 31 02:06:53.089252 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.089231 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.145002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.144969 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg"] Oct 31 02:06:53.223488 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.223454 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:06:53.223681 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.223520 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:06:53.243949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.243912 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjpc2\" (UniqueName: \"kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.244065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.243973 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.244136 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.244087 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.345728 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.345637 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.345728 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.345713 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.346261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.345927 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjpc2\" (UniqueName: \"kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.346391 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.346369 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.348527 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.348500 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.404145 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.404122 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjpc2\" (UniqueName: \"kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2\") pod \"maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:53.702466 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:53.702387 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:54.080243 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:54.080220 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg"] Oct 31 02:06:54.082368 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:06:54.082341 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0ed839_7e95_427c_b842_9397caf0d32c.slice/crio-96189b53b8aec4ef4fefdccdf5e472a22bfef45c4249411782cffc6fb5302e2a WatchSource:0}: Error finding container 96189b53b8aec4ef4fefdccdf5e472a22bfef45c4249411782cffc6fb5302e2a: Status 404 returned error can't find the container with id 96189b53b8aec4ef4fefdccdf5e472a22bfef45c4249411782cffc6fb5302e2a Oct 31 02:06:54.151991 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:54.151959 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" event={"ID":"bf0ed839-7e95-427c-b842-9397caf0d32c","Type":"ContainerStarted","Data":"96189b53b8aec4ef4fefdccdf5e472a22bfef45c4249411782cffc6fb5302e2a"} Oct 31 02:06:54.312508 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:54.312476 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" path="/var/lib/kubelet/pods/8915ddc2-27bd-4d6e-a6ec-c6d318371863/volumes" Oct 31 02:06:55.160443 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:55.160355 5756 generic.go:356] "Generic (PLEG): container finished" podID="bf0ed839-7e95-427c-b842-9397caf0d32c" containerID="1003daa7d4a002a03460dbabca312707aa86ffd5ee9b25e9692af55377c2762e" exitCode=0 Oct 31 02:06:55.160815 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:55.160435 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" event={"ID":"bf0ed839-7e95-427c-b842-9397caf0d32c","Type":"ContainerDied","Data":"1003daa7d4a002a03460dbabca312707aa86ffd5ee9b25e9692af55377c2762e"} Oct 31 02:06:56.322428 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.322400 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:56.486665 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.486590 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2\" (UniqueName: \"kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2\") pod \"bf0ed839-7e95-427c-b842-9397caf0d32c\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " Oct 31 02:06:56.486665 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.486660 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca\") pod \"bf0ed839-7e95-427c-b842-9397caf0d32c\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " Oct 31 02:06:56.486855 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.486806 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca\") pod \"bf0ed839-7e95-427c-b842-9397caf0d32c\" (UID: \"bf0ed839-7e95-427c-b842-9397caf0d32c\") " Oct 31 02:06:56.487084 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.487049 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca" pod "bf0ed839-7e95-427c-b842-9397caf0d32c" (UID: "bf0ed839-7e95-427c-b842-9397caf0d32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:06:56.487195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.487181 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:56.488979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.488957 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca" pod "bf0ed839-7e95-427c-b842-9397caf0d32c" (UID: "bf0ed839-7e95-427c-b842-9397caf0d32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:06:56.489080 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.489022 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2" pod "bf0ed839-7e95-427c-b842-9397caf0d32c" (UID: "bf0ed839-7e95-427c-b842-9397caf0d32c"). InnerVolumeSpecName "kube-api-access-wjpc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:06:56.588354 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.588313 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0ed839-7e95-427c-b842-9397caf0d32c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:56.588354 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:56.588356 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjpc2\" (UniqueName: \"kubernetes.io/projected/bf0ed839-7e95-427c-b842-9397caf0d32c-kube-api-access-wjpc2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:06:57.174389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:57.174359 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" Oct 31 02:06:57.174389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:57.174383 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg" event={"ID":"bf0ed839-7e95-427c-b842-9397caf0d32c","Type":"ContainerDied","Data":"96189b53b8aec4ef4fefdccdf5e472a22bfef45c4249411782cffc6fb5302e2a"} Oct 31 02:06:57.174645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:57.174431 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96189b53b8aec4ef4fefdccdf5e472a22bfef45c4249411782cffc6fb5302e2a" Oct 31 02:06:58.349581 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:58.349517 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw"] Oct 31 02:06:58.350015 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:58.349785 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" containerID="cri-o://4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f" gracePeriod=30 Oct 31 02:06:59.563736 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.563704 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz"] Oct 31 02:06:59.564348 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.564331 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bf0ed839-7e95-427c-b842-9397caf0d32c" containerName="revoke-keys" Oct 31 02:06:59.564416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.564351 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0ed839-7e95-427c-b842-9397caf0d32c" containerName="revoke-keys" Oct 31 02:06:59.564416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.564364 5756 cpu_manager.go:414] "containerMap: removing container" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" Oct 31 02:06:59.564416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.564369 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8915ddc2-27bd-4d6e-a6ec-c6d318371863" containerName="istio-proxy" Oct 31 02:06:59.564513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.564461 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0ed839-7e95-427c-b842-9397caf0d32c" containerName="revoke-keys" Oct 31 02:06:59.568340 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.568321 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.623827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.623796 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz"] Oct 31 02:06:59.633140 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.633107 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbg9\" (UniqueName: \"kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.633299 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.633281 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.633368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.633352 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.669248 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.669220 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:06:59.669248 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.669233 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:06:59.734705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.734672 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.734894 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.734755 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.734894 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.734803 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbg9\" (UniqueName: \"kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.735482 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.735456 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.737231 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.737208 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.791199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.791166 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbg9\" (UniqueName: \"kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9\") pod \"maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:06:59.881512 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:06:59.881430 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:07:00.260315 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:00.260291 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz"] Oct 31 02:07:00.261878 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:07:00.261831 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9d3035_3efe_4a38_9254_6c69b622ef5d.slice/crio-91754b79b417a01d9ebaf500e158ee491e2e11c095700ec95ff42422d543c9b9 WatchSource:0}: Error finding container 91754b79b417a01d9ebaf500e158ee491e2e11c095700ec95ff42422d543c9b9: Status 404 returned error can't find the container with id 91754b79b417a01d9ebaf500e158ee491e2e11c095700ec95ff42422d543c9b9 Oct 31 02:07:01.205420 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:01.205384 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" event={"ID":"0a9d3035-3efe-4a38-9254-6c69b622ef5d","Type":"ContainerStarted","Data":"41dbe3e2d32fe00e164e4e977375796405f7f132b95669eeb087fefddd41b95f"} Oct 31 02:07:01.205903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:01.205464 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" event={"ID":"0a9d3035-3efe-4a38-9254-6c69b622ef5d","Type":"ContainerStarted","Data":"91754b79b417a01d9ebaf500e158ee491e2e11c095700ec95ff42422d543c9b9"} Oct 31 02:07:01.318438 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:01.318368 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" podStartSLOduration=2.31835402 podStartE2EDuration="2.31835402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:01.316616604 +0000 UTC m=+4193.123584160" watchObservedRunningTime="2026-10-31 02:07:01.31835402 +0000 UTC m=+4193.125321563" Oct 31 02:07:02.068249 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:07:02.068208 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:07:02.213356 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:02.213320 5756 generic.go:356] "Generic (PLEG): container finished" podID="0a9d3035-3efe-4a38-9254-6c69b622ef5d" containerID="41dbe3e2d32fe00e164e4e977375796405f7f132b95669eeb087fefddd41b95f" exitCode=0 Oct 31 02:07:02.213746 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:02.213403 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" event={"ID":"0a9d3035-3efe-4a38-9254-6c69b622ef5d","Type":"ContainerDied","Data":"41dbe3e2d32fe00e164e4e977375796405f7f132b95669eeb087fefddd41b95f"} Oct 31 02:07:02.604943 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:02.604906 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:07:02.605126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:02.604968 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:07:03.446009 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.445982 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:07:03.483220 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.483133 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca\") pod \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " Oct 31 02:07:03.483374 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.483342 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca\") pod \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " Oct 31 02:07:03.483458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.483445 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9\" (UniqueName: \"kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9\") pod \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\" (UID: \"0a9d3035-3efe-4a38-9254-6c69b622ef5d\") " Oct 31 02:07:03.483757 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.483726 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca" pod "0a9d3035-3efe-4a38-9254-6c69b622ef5d" (UID: "0a9d3035-3efe-4a38-9254-6c69b622ef5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:03.483998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.483944 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.485719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.485690 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9" pod "0a9d3035-3efe-4a38-9254-6c69b622ef5d" (UID: "0a9d3035-3efe-4a38-9254-6c69b622ef5d"). InnerVolumeSpecName "kube-api-access-zgbg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:03.485719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.485709 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca" pod "0a9d3035-3efe-4a38-9254-6c69b622ef5d" (UID: "0a9d3035-3efe-4a38-9254-6c69b622ef5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:07:03.585793 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.585750 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgbg9\" (UniqueName: \"kubernetes.io/projected/0a9d3035-3efe-4a38-9254-6c69b622ef5d-kube-api-access-zgbg9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.585793 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.585791 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a9d3035-3efe-4a38-9254-6c69b622ef5d-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.615759 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.615736 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 02:07:03.686159 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686134 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686238 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686258 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686302 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686428 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686381 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686428 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686415 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686528 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686467 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686590 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686540 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:03.686590 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686571 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686579 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:03.686705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686602 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:03.686705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686622 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884\") pod \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\" (UID: \"3e05eef2-76d3-4dec-bea6-bef86f3fdf31\") " Oct 31 02:07:03.686899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686738 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:03.686899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.686808 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:03.687409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.687337 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.687409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.687365 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.687409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.687381 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.687409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.687397 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.687409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.687413 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.688639 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.688614 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:03.688741 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.688708 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:03.688801 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.688769 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:07:03.688989 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.688974 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884" pod "3e05eef2-76d3-4dec-bea6-bef86f3fdf31" (UID: "3e05eef2-76d3-4dec-bea6-bef86f3fdf31"). InnerVolumeSpecName "kube-api-access-zl884". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:03.788797 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.788763 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.788797 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.788795 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl884\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-kube-api-access-zl884\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.789019 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.788806 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:03.789019 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:03.788816 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e05eef2-76d3-4dec-bea6-bef86f3fdf31-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:04.228706 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.228625 5756 generic.go:356] "Generic (PLEG): container finished" podID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerID="4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f" exitCode=0 Oct 31 02:07:04.228706 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.228675 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" Oct 31 02:07:04.228983 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.228704 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" event={"ID":"3e05eef2-76d3-4dec-bea6-bef86f3fdf31","Type":"ContainerDied","Data":"4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f"} Oct 31 02:07:04.228983 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.228759 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw" event={"ID":"3e05eef2-76d3-4dec-bea6-bef86f3fdf31","Type":"ContainerDied","Data":"2b247285831c0a485f090774604b44c54e7c37ef564f46e27fc214dea82c6f13"} Oct 31 02:07:04.228983 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.228788 5756 scope.go:122] "RemoveContainer" containerID="4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f" Oct 31 02:07:04.230114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.230099 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" Oct 31 02:07:04.230185 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.230160 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz" event={"ID":"0a9d3035-3efe-4a38-9254-6c69b622ef5d","Type":"ContainerDied","Data":"91754b79b417a01d9ebaf500e158ee491e2e11c095700ec95ff42422d543c9b9"} Oct 31 02:07:04.230226 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.230204 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91754b79b417a01d9ebaf500e158ee491e2e11c095700ec95ff42422d543c9b9" Oct 31 02:07:04.240854 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.240838 5756 scope.go:122] "RemoveContainer" containerID="285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90" Oct 31 02:07:04.250181 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.250164 5756 scope.go:122] "RemoveContainer" containerID="4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f" Oct 31 02:07:04.250466 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:07:04.250436 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f\": container with ID starting with 4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f not found: ID does not exist" containerID="4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f" Oct 31 02:07:04.250515 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.250467 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f"} err="failed to get container status \"4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f\": rpc error: code = NotFound desc = could not find container \"4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f\": container with ID starting with 4e546f3411f78f82117e79e78fb5909d0d0e99d774bb7a886de1f1edf7fb0c8f not found: ID does not exist" Oct 31 02:07:04.250515 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.250486 5756 scope.go:122] "RemoveContainer" containerID="285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90" Oct 31 02:07:04.250735 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:07:04.250718 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90\": container with ID starting with 285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90 not found: ID does not exist" containerID="285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90" Oct 31 02:07:04.250780 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.250743 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90"} err="failed to get container status \"285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90\": rpc error: code = NotFound desc = could not find container \"285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90\": container with ID starting with 285c18325112d6d467b5da33f37a0f07f063288d04e8be0935fe0df549617e90 not found: ID does not exist" Oct 31 02:07:04.393788 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.393752 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw"] Oct 31 02:07:04.404952 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:04.404927 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6caf19-openshift-default-66c887d577-8m9qw"] Oct 31 02:07:06.311387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:06.311346 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" path="/var/lib/kubelet/pods/3e05eef2-76d3-4dec-bea6-bef86f3fdf31/volumes" Oct 31 02:07:06.968969 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:06.968933 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" podUID="99c1d371-db3f-4b9f-b4e5-d2a709b4486a" containerName="main" containerID="cri-o://762cb180d117120ea671fa3284a808202c708ccf0ccfdcaadc9a8fe336868a52" gracePeriod=45 Oct 31 02:07:06.971584 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:06.971556 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerName="main" containerID="cri-o://9187859790d0f45e09bf7060c86104419d5d239edad97acbc8bc4e5b41a22fb7" gracePeriod=45 Oct 31 02:07:06.983655 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:06.983627 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" containerID="cri-o://a1634d2bf410afd1d6d7823998ccb3892b5b47185346eee721820e24154a0c7e" gracePeriod=45 Oct 31 02:07:07.024357 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.024297 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 31 02:07:07.257330 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.257300 5756 generic.go:356] "Generic (PLEG): container finished" podID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerID="9187859790d0f45e09bf7060c86104419d5d239edad97acbc8bc4e5b41a22fb7" exitCode=0 Oct 31 02:07:07.257454 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.257436 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" event={"ID":"92852b04-1bb0-4c84-8c7f-8ea8feec9f50","Type":"ContainerDied","Data":"9187859790d0f45e09bf7060c86104419d5d239edad97acbc8bc4e5b41a22fb7"} Oct 31 02:07:07.259292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.259265 5756 generic.go:356] "Generic (PLEG): container finished" podID="966522e7-caaf-4514-abd9-8b9111b971c0" containerID="a1634d2bf410afd1d6d7823998ccb3892b5b47185346eee721820e24154a0c7e" exitCode=0 Oct 31 02:07:07.259398 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.259355 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" event={"ID":"966522e7-caaf-4514-abd9-8b9111b971c0","Type":"ContainerDied","Data":"a1634d2bf410afd1d6d7823998ccb3892b5b47185346eee721820e24154a0c7e"} Oct 31 02:07:07.261192 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.261166 5756 generic.go:356] "Generic (PLEG): container finished" podID="99c1d371-db3f-4b9f-b4e5-d2a709b4486a" containerID="762cb180d117120ea671fa3284a808202c708ccf0ccfdcaadc9a8fe336868a52" exitCode=0 Oct 31 02:07:07.261299 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.261203 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" event={"ID":"99c1d371-db3f-4b9f-b4e5-d2a709b4486a","Type":"ContainerDied","Data":"762cb180d117120ea671fa3284a808202c708ccf0ccfdcaadc9a8fe336868a52"} Oct 31 02:07:07.273268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.273246 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 02:07:07.307658 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.306085 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 02:07:07.311305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.311276 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 02:07:07.333375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.333341 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache\") pod \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " Oct 31 02:07:07.333934 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.333393 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir\") pod \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " Oct 31 02:07:07.333934 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.333422 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm\") pod \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " Oct 31 02:07:07.333934 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.333501 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl\" (UniqueName: \"kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl\") pod \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " Oct 31 02:07:07.333934 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.333542 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home\") pod \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " Oct 31 02:07:07.333934 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.333634 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs\" (UniqueName: \"kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs\") pod \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\" (UID: \"92852b04-1bb0-4c84-8c7f-8ea8feec9f50\") " Oct 31 02:07:07.335011 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.334965 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir" pod "92852b04-1bb0-4c84-8c7f-8ea8feec9f50" (UID: "92852b04-1bb0-4c84-8c7f-8ea8feec9f50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.335011 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.334977 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache" pod "92852b04-1bb0-4c84-8c7f-8ea8feec9f50" (UID: "92852b04-1bb0-4c84-8c7f-8ea8feec9f50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.335188 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.335005 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home" pod "92852b04-1bb0-4c84-8c7f-8ea8feec9f50" (UID: "92852b04-1bb0-4c84-8c7f-8ea8feec9f50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.337776 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.337744 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl" pod "92852b04-1bb0-4c84-8c7f-8ea8feec9f50" (UID: "92852b04-1bb0-4c84-8c7f-8ea8feec9f50"). InnerVolumeSpecName "kube-api-access-szbwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:07.337920 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.337828 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs" pod "92852b04-1bb0-4c84-8c7f-8ea8feec9f50" (UID: "92852b04-1bb0-4c84-8c7f-8ea8feec9f50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:07:07.338121 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.338105 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm" pod "92852b04-1bb0-4c84-8c7f-8ea8feec9f50" (UID: "92852b04-1bb0-4c84-8c7f-8ea8feec9f50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.434679 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434647 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs\") pod \"966522e7-caaf-4514-abd9-8b9111b971c0\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " Oct 31 02:07:07.434903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434710 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir\") pod \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " Oct 31 02:07:07.434903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434755 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache\") pod \"966522e7-caaf-4514-abd9-8b9111b971c0\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " Oct 31 02:07:07.434903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434815 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4\" (UniqueName: \"kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4\") pod \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " Oct 31 02:07:07.434903 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434851 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs\") pod \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " Oct 31 02:07:07.435158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434910 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm\") pod \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " Oct 31 02:07:07.435158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434944 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home\") pod \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " Oct 31 02:07:07.435158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.434974 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache\") pod \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\" (UID: \"99c1d371-db3f-4b9f-b4e5-d2a709b4486a\") " Oct 31 02:07:07.435158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435018 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm\") pod \"966522e7-caaf-4514-abd9-8b9111b971c0\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " Oct 31 02:07:07.435158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435090 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p\" (UniqueName: \"kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p\") pod \"966522e7-caaf-4514-abd9-8b9111b971c0\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " Oct 31 02:07:07.435158 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435140 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir\") pod \"966522e7-caaf-4514-abd9-8b9111b971c0\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " Oct 31 02:07:07.435485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435167 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache" pod "966522e7-caaf-4514-abd9-8b9111b971c0" (UID: "966522e7-caaf-4514-abd9-8b9111b971c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.435485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435178 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home\") pod \"966522e7-caaf-4514-abd9-8b9111b971c0\" (UID: \"966522e7-caaf-4514-abd9-8b9111b971c0\") " Oct 31 02:07:07.435485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435340 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home" pod "966522e7-caaf-4514-abd9-8b9111b971c0" (UID: "966522e7-caaf-4514-abd9-8b9111b971c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.435662 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435523 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache" pod "99c1d371-db3f-4b9f-b4e5-d2a709b4486a" (UID: "99c1d371-db3f-4b9f-b4e5-d2a709b4486a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.435748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435721 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home" pod "99c1d371-db3f-4b9f-b4e5-d2a709b4486a" (UID: "99c1d371-db3f-4b9f-b4e5-d2a709b4486a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.435927 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.435833 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.436109 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436091 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.436229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436214 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.436333 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436320 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.436434 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436420 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.436553 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436541 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.436697 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436683 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.437067 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436373 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir" pod "99c1d371-db3f-4b9f-b4e5-d2a709b4486a" (UID: "99c1d371-db3f-4b9f-b4e5-d2a709b4486a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.437067 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436427 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir" pod "966522e7-caaf-4514-abd9-8b9111b971c0" (UID: "966522e7-caaf-4514-abd9-8b9111b971c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.437241 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.436943 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szbwl\" (UniqueName: \"kubernetes.io/projected/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-kube-api-access-szbwl\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.437343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.437328 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.437442 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.437431 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92852b04-1bb0-4c84-8c7f-8ea8feec9f50-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.438181 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.438156 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs" pod "966522e7-caaf-4514-abd9-8b9111b971c0" (UID: "966522e7-caaf-4514-abd9-8b9111b971c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:07:07.438563 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.438529 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs" pod "99c1d371-db3f-4b9f-b4e5-d2a709b4486a" (UID: "99c1d371-db3f-4b9f-b4e5-d2a709b4486a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:07:07.439351 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.439326 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm" pod "99c1d371-db3f-4b9f-b4e5-d2a709b4486a" (UID: "99c1d371-db3f-4b9f-b4e5-d2a709b4486a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.439531 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.439485 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm" pod "966522e7-caaf-4514-abd9-8b9111b971c0" (UID: "966522e7-caaf-4514-abd9-8b9111b971c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:07:07.439635 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.439616 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p" pod "966522e7-caaf-4514-abd9-8b9111b971c0" (UID: "966522e7-caaf-4514-abd9-8b9111b971c0"). InnerVolumeSpecName "kube-api-access-x2t2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:07.440742 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.440716 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4" pod "99c1d371-db3f-4b9f-b4e5-d2a709b4486a" (UID: "99c1d371-db3f-4b9f-b4e5-d2a709b4486a"). InnerVolumeSpecName "kube-api-access-wj9c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:07.538963 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.538935 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966522e7-caaf-4514-abd9-8b9111b971c0-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.538963 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.538963 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.539218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.538973 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj9c4\" (UniqueName: \"kubernetes.io/projected/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-kube-api-access-wj9c4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.539218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.538982 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.539218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.538990 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1d371-db3f-4b9f-b4e5-d2a709b4486a-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.539218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.538998 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.539218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.539005 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2t2p\" (UniqueName: \"kubernetes.io/projected/966522e7-caaf-4514-abd9-8b9111b971c0-kube-api-access-x2t2p\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:07.539218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:07.539013 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966522e7-caaf-4514-abd9-8b9111b971c0-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:08.261100 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.261013 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 02:07:08.268161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.268141 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" Oct 31 02:07:08.268299 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.268185 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92" event={"ID":"92852b04-1bb0-4c84-8c7f-8ea8feec9f50","Type":"ContainerDied","Data":"c35728f41ce2c9745097388c1d67e9ef30ce4050695da25121d4ad52501b4776"} Oct 31 02:07:08.268299 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.268233 5756 scope.go:122] "RemoveContainer" containerID="9187859790d0f45e09bf7060c86104419d5d239edad97acbc8bc4e5b41a22fb7" Oct 31 02:07:08.269753 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.269731 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" Oct 31 02:07:08.269753 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.269741 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz" event={"ID":"966522e7-caaf-4514-abd9-8b9111b971c0","Type":"ContainerDied","Data":"35ce312eae781f51af40922f69cc5bf39ce18320c7503ff7dc056f589741b56d"} Oct 31 02:07:08.271631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.271602 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" Oct 31 02:07:08.271748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.271645 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q" event={"ID":"99c1d371-db3f-4b9f-b4e5-d2a709b4486a","Type":"ContainerDied","Data":"1169ae85c5b384487c59b2b6f70dfd425beb27c249fa731cbf32a2853a1c7b69"} Oct 31 02:07:08.278984 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.278967 5756 scope.go:122] "RemoveContainer" containerID="a1634d2bf410afd1d6d7823998ccb3892b5b47185346eee721820e24154a0c7e" Oct 31 02:07:08.288161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.288143 5756 scope.go:122] "RemoveContainer" containerID="762cb180d117120ea671fa3284a808202c708ccf0ccfdcaadc9a8fe336868a52" Oct 31 02:07:08.517849 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.517756 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92"] Oct 31 02:07:08.520539 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.520515 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-w0-26fbc4-kserve-9855959d-jgm92"] Oct 31 02:07:08.631967 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.631931 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz"] Oct 31 02:07:08.635745 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.635718 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/premium-w0-26fbc4-kserve-5d988688f9-l75bz"] Oct 31 02:07:08.742803 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.742769 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q"] Oct 31 02:07:08.746494 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:08.746466 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-26fbc4/distinct-2-w0-26fbc4-kserve-86bcb9b67c-5nh2q"] Oct 31 02:07:10.312814 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:10.312778 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" path="/var/lib/kubelet/pods/92852b04-1bb0-4c84-8c7f-8ea8feec9f50/volumes" Oct 31 02:07:10.313291 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:10.313275 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" path="/var/lib/kubelet/pods/966522e7-caaf-4514-abd9-8b9111b971c0/volumes" Oct 31 02:07:10.313698 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:10.313685 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c1d371-db3f-4b9f-b4e5-d2a709b4486a" path="/var/lib/kubelet/pods/99c1d371-db3f-4b9f-b4e5-d2a709b4486a/volumes" Oct 31 02:07:11.025909 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:11.025854 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb"] Oct 31 02:07:11.135911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:11.135847 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2"] Oct 31 02:07:11.352902 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:11.351799 5756 scope.go:122] "RemoveContainer" containerID="d59fa54d590218f3d793fd653902f14a62e2ea65bcdee8afe692de6216685516" Oct 31 02:07:11.374230 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:11.374207 5756 scope.go:122] "RemoveContainer" containerID="a320d401354084a922fd394581a87b0cbc9dde11fd1e113b192740f4c2aaf585" Oct 31 02:07:13.373740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:13.373707 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm"] Oct 31 02:07:13.424872 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:13.424829 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ce8dbb-d2fwm"] Oct 31 02:07:14.311220 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:14.311182 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260d9a2e-59b3-4468-bce8-3a8c69f817d6" path="/var/lib/kubelet/pods/260d9a2e-59b3-4468-bce8-3a8c69f817d6/volumes" Oct 31 02:07:16.026902 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.026843 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" podUID="6fa805c7-f24b-489a-9629-689a4f596721" containerName="payload-processing" containerID="cri-o://535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446" gracePeriod=25 Oct 31 02:07:16.136996 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.136960 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" podUID="c238dd99-399b-4dd2-b9cf-520021f63177" containerName="payload-pre-processing" containerID="cri-o://aa1523623f52ac5919eba2def9ddfa155204261908a024e28eb3601e0826184c" gracePeriod=25 Oct 31 02:07:16.326228 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.326183 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:07:16.335596 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.335568 5756 generic.go:356] "Generic (PLEG): container finished" podID="c238dd99-399b-4dd2-b9cf-520021f63177" containerID="aa1523623f52ac5919eba2def9ddfa155204261908a024e28eb3601e0826184c" exitCode=0 Oct 31 02:07:16.335732 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.335633 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" event={"ID":"c238dd99-399b-4dd2-b9cf-520021f63177","Type":"ContainerDied","Data":"aa1523623f52ac5919eba2def9ddfa155204261908a024e28eb3601e0826184c"} Oct 31 02:07:16.337457 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.337431 5756 generic.go:356] "Generic (PLEG): container finished" podID="6fa805c7-f24b-489a-9629-689a4f596721" containerID="535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446" exitCode=0 Oct 31 02:07:16.337575 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.337507 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" event={"ID":"6fa805c7-f24b-489a-9629-689a4f596721","Type":"ContainerDied","Data":"535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446"} Oct 31 02:07:16.337575 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.337541 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" Oct 31 02:07:16.337575 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.337548 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb" event={"ID":"6fa805c7-f24b-489a-9629-689a4f596721","Type":"ContainerDied","Data":"d3695f3d3525a7ff6cc41818fcc13eefa91aa7d83e4f1959054f2d04ad14e57c"} Oct 31 02:07:16.337748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.337579 5756 scope.go:122] "RemoveContainer" containerID="535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446" Oct 31 02:07:16.349958 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.349507 5756 scope.go:122] "RemoveContainer" containerID="535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446" Oct 31 02:07:16.349958 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:07:16.349845 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446\": container with ID starting with 535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446 not found: ID does not exist" containerID="535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446" Oct 31 02:07:16.349958 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.349905 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446"} err="failed to get container status \"535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446\": rpc error: code = NotFound desc = could not find container \"535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446\": container with ID starting with 535ae6ac7df9607ed0b8574876a515534550aeecabe1c37e3d92365c1ac02446 not found: ID does not exist" Oct 31 02:07:16.435286 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.435259 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:07:16.445290 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.445264 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp\" (UniqueName: \"kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp\") pod \"6fa805c7-f24b-489a-9629-689a4f596721\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " Oct 31 02:07:16.445461 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.445345 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume\") pod \"6fa805c7-f24b-489a-9629-689a4f596721\" (UID: \"6fa805c7-f24b-489a-9629-689a4f596721\") " Oct 31 02:07:16.445758 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.445728 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume" pod "6fa805c7-f24b-489a-9629-689a4f596721" (UID: "6fa805c7-f24b-489a-9629-689a4f596721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:16.445901 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.445885 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa805c7-f24b-489a-9629-689a4f596721-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:16.447565 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.447542 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp" pod "6fa805c7-f24b-489a-9629-689a4f596721" (UID: "6fa805c7-f24b-489a-9629-689a4f596721"). InnerVolumeSpecName "kube-api-access-6qvcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:16.546832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.546739 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume\") pod \"c238dd99-399b-4dd2-b9cf-520021f63177\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " Oct 31 02:07:16.546832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.546820 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq\" (UniqueName: \"kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq\") pod \"c238dd99-399b-4dd2-b9cf-520021f63177\" (UID: \"c238dd99-399b-4dd2-b9cf-520021f63177\") " Oct 31 02:07:16.547173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.547151 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume" pod "c238dd99-399b-4dd2-b9cf-520021f63177" (UID: "c238dd99-399b-4dd2-b9cf-520021f63177"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:16.547275 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.547261 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c238dd99-399b-4dd2-b9cf-520021f63177-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:16.547326 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.547278 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvcp\" (UniqueName: \"kubernetes.io/projected/6fa805c7-f24b-489a-9629-689a4f596721-kube-api-access-6qvcp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:16.549066 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.549045 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq" pod "c238dd99-399b-4dd2-b9cf-520021f63177" (UID: "c238dd99-399b-4dd2-b9cf-520021f63177"). InnerVolumeSpecName "kube-api-access-5ntgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:16.648378 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.648351 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ntgq\" (UniqueName: \"kubernetes.io/projected/c238dd99-399b-4dd2-b9cf-520021f63177-kube-api-access-5ntgq\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:16.759247 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.759214 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb"] Oct 31 02:07:16.763545 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:16.763518 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-8a35f7-594c6b5865-mnvjb"] Oct 31 02:07:17.347243 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:17.347165 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" Oct 31 02:07:17.347243 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:17.347201 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2" event={"ID":"c238dd99-399b-4dd2-b9cf-520021f63177","Type":"ContainerDied","Data":"cefb00428058a69f294a98bfefb29218c2542f12820ef6cd4e3bac9f8076d927"} Oct 31 02:07:17.347783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:17.347263 5756 scope.go:122] "RemoveContainer" containerID="aa1523623f52ac5919eba2def9ddfa155204261908a024e28eb3601e0826184c" Oct 31 02:07:17.518070 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:17.518034 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2"] Oct 31 02:07:17.530992 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:17.530959 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-8a35f7-7f776c4495-nmrh2"] Oct 31 02:07:18.314579 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:18.314550 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa805c7-f24b-489a-9629-689a4f596721" path="/var/lib/kubelet/pods/6fa805c7-f24b-489a-9629-689a4f596721/volumes" Oct 31 02:07:18.314950 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:18.314935 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c238dd99-399b-4dd2-b9cf-520021f63177" path="/var/lib/kubelet/pods/c238dd99-399b-4dd2-b9cf-520021f63177/volumes" Oct 31 02:07:20.747845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.747807 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr"] Oct 31 02:07:20.748404 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748390 5756 cpu_manager.go:414] "containerMap: removing container" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" Oct 31 02:07:20.748455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748406 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" Oct 31 02:07:20.748455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748431 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6fa805c7-f24b-489a-9629-689a4f596721" containerName="payload-processing" Oct 31 02:07:20.748455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748436 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa805c7-f24b-489a-9629-689a4f596721" containerName="payload-processing" Oct 31 02:07:20.748455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748446 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0a9d3035-3efe-4a38-9254-6c69b622ef5d" containerName="revoke-keys" Oct 31 02:07:20.748455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748453 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a9d3035-3efe-4a38-9254-6c69b622ef5d" containerName="revoke-keys" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748462 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748467 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748477 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c238dd99-399b-4dd2-b9cf-520021f63177" containerName="payload-pre-processing" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748482 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c238dd99-399b-4dd2-b9cf-520021f63177" containerName="payload-pre-processing" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748490 5756 cpu_manager.go:414] "containerMap: removing container" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerName="main" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748495 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerName="main" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748508 5756 cpu_manager.go:414] "containerMap: removing container" podUID="99c1d371-db3f-4b9f-b4e5-d2a709b4486a" containerName="main" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748512 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c1d371-db3f-4b9f-b4e5-d2a709b4486a" containerName="main" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748518 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748522 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748598 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa805c7-f24b-489a-9629-689a4f596721" containerName="payload-processing" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748607 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a9d3035-3efe-4a38-9254-6c69b622ef5d" containerName="revoke-keys" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748613 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748622 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c238dd99-399b-4dd2-b9cf-520021f63177" containerName="payload-pre-processing" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748627 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="92852b04-1bb0-4c84-8c7f-8ea8feec9f50" containerName="main" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748634 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c1d371-db3f-4b9f-b4e5-d2a709b4486a" containerName="main" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748639 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e05eef2-76d3-4dec-bea6-bef86f3fdf31" containerName="istio-proxy" Oct 31 02:07:20.748647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.748646 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="966522e7-caaf-4514-abd9-8b9111b971c0" containerName="main" Oct 31 02:07:20.753339 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.753317 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.854614 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.854580 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:07:20.854790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.854614 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:07:20.857198 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.857178 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr"] Oct 31 02:07:20.895389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.895361 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.895513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.895495 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.895575 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.895530 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6sn5\" (UniqueName: \"kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.997222 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.997188 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.997366 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.997231 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6sn5\" (UniqueName: \"kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.997366 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.997316 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.997827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.997793 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:20.999761 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:20.999720 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:21.054651 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:21.054622 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6sn5\" (UniqueName: \"kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5\") pod \"maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:21.066502 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:21.066478 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:21.441801 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:21.441781 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr"] Oct 31 02:07:21.443759 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:07:21.443733 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92b5f68_53bc_4f91_85ea_0677756c6066.slice/crio-d6505a2e7129a661983d98d11dc48b60a1b0d6433686966f355d3a36ecd36f4c WatchSource:0}: Error finding container d6505a2e7129a661983d98d11dc48b60a1b0d6433686966f355d3a36ecd36f4c: Status 404 returned error can't find the container with id d6505a2e7129a661983d98d11dc48b60a1b0d6433686966f355d3a36ecd36f4c Oct 31 02:07:22.381015 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:22.380982 5756 generic.go:356] "Generic (PLEG): container finished" podID="f92b5f68-53bc-4f91-85ea-0677756c6066" containerID="ed465996dc53f3050b923b987d0414577ef89c1f359a788f9a376c141570edf6" exitCode=0 Oct 31 02:07:22.381345 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:22.381067 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" event={"ID":"f92b5f68-53bc-4f91-85ea-0677756c6066","Type":"ContainerDied","Data":"ed465996dc53f3050b923b987d0414577ef89c1f359a788f9a376c141570edf6"} Oct 31 02:07:22.381345 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:22.381119 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" event={"ID":"f92b5f68-53bc-4f91-85ea-0677756c6066","Type":"ContainerStarted","Data":"d6505a2e7129a661983d98d11dc48b60a1b0d6433686966f355d3a36ecd36f4c"} Oct 31 02:07:23.548050 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.548027 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:23.726943 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.726825 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca\") pod \"f92b5f68-53bc-4f91-85ea-0677756c6066\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " Oct 31 02:07:23.727135 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.727006 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5\" (UniqueName: \"kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5\") pod \"f92b5f68-53bc-4f91-85ea-0677756c6066\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " Oct 31 02:07:23.727135 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.727080 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca\") pod \"f92b5f68-53bc-4f91-85ea-0677756c6066\" (UID: \"f92b5f68-53bc-4f91-85ea-0677756c6066\") " Oct 31 02:07:23.727273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.727243 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca" pod "f92b5f68-53bc-4f91-85ea-0677756c6066" (UID: "f92b5f68-53bc-4f91-85ea-0677756c6066"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:23.727487 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.727471 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:23.729258 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.729227 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5" pod "f92b5f68-53bc-4f91-85ea-0677756c6066" (UID: "f92b5f68-53bc-4f91-85ea-0677756c6066"). InnerVolumeSpecName "kube-api-access-r6sn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:23.729352 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.729267 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca" pod "f92b5f68-53bc-4f91-85ea-0677756c6066" (UID: "f92b5f68-53bc-4f91-85ea-0677756c6066"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:07:23.828358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.828326 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6sn5\" (UniqueName: \"kubernetes.io/projected/f92b5f68-53bc-4f91-85ea-0677756c6066-kube-api-access-r6sn5\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:23.828358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:23.828360 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f92b5f68-53bc-4f91-85ea-0677756c6066-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:24.394493 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:24.394465 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" Oct 31 02:07:24.394493 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:24.394475 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr" event={"ID":"f92b5f68-53bc-4f91-85ea-0677756c6066","Type":"ContainerDied","Data":"d6505a2e7129a661983d98d11dc48b60a1b0d6433686966f355d3a36ecd36f4c"} Oct 31 02:07:24.394699 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:24.394517 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6505a2e7129a661983d98d11dc48b60a1b0d6433686966f355d3a36ecd36f4c" Oct 31 02:07:30.016829 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:30.016785 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz"] Oct 31 02:07:30.018895 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:30.018854 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-8a35f7-d5ppz"] Oct 31 02:07:30.311408 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:30.311376 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a9d3035-3efe-4a38-9254-6c69b622ef5d" path="/var/lib/kubelet/pods/0a9d3035-3efe-4a38-9254-6c69b622ef5d/volumes" Oct 31 02:07:32.067301 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:07:32.067263 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:07:37.773021 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.772988 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw"] Oct 31 02:07:37.773507 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.773501 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f92b5f68-53bc-4f91-85ea-0677756c6066" containerName="revoke-keys" Oct 31 02:07:37.773586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.773512 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f92b5f68-53bc-4f91-85ea-0677756c6066" containerName="revoke-keys" Oct 31 02:07:37.773642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.773628 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f92b5f68-53bc-4f91-85ea-0677756c6066" containerName="revoke-keys" Oct 31 02:07:37.778074 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.778055 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:37.880122 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.880086 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:07:37.880122 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.880098 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:07:37.884204 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.884179 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw"] Oct 31 02:07:37.900277 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.900242 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:37.900522 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.900286 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:37.900522 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:37.900366 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbm9p\" (UniqueName: \"kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.001762 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.001723 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.001971 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.001779 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.001971 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.001837 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbm9p\" (UniqueName: \"kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.002586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.002557 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.004730 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.004693 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.058263 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.058227 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbm9p\" (UniqueName: \"kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.091012 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.090972 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:38.469697 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.469676 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw"] Oct 31 02:07:38.472123 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:07:38.472083 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fcff7a_4655_438b_aa95_b4b95c0ce152.slice/crio-e6049dda7bd3687310422a31b6d32861a396014721db98012e7d2bba6e5bfc0a WatchSource:0}: Error finding container e6049dda7bd3687310422a31b6d32861a396014721db98012e7d2bba6e5bfc0a: Status 404 returned error can't find the container with id e6049dda7bd3687310422a31b6d32861a396014721db98012e7d2bba6e5bfc0a Oct 31 02:07:38.497452 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:38.497421 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" event={"ID":"22fcff7a-4655-438b-aa95-b4b95c0ce152","Type":"ContainerStarted","Data":"e6049dda7bd3687310422a31b6d32861a396014721db98012e7d2bba6e5bfc0a"} Oct 31 02:07:39.506279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:39.506248 5756 generic.go:356] "Generic (PLEG): container finished" podID="22fcff7a-4655-438b-aa95-b4b95c0ce152" containerID="f74eb08716fdc8defcfc03105bb2763e8392f0cd0d0d2bd07ab69f2609f6e874" exitCode=0 Oct 31 02:07:39.506739 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:39.506348 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" event={"ID":"22fcff7a-4655-438b-aa95-b4b95c0ce152","Type":"ContainerDied","Data":"f74eb08716fdc8defcfc03105bb2763e8392f0cd0d0d2bd07ab69f2609f6e874"} Oct 31 02:07:40.650552 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.650523 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:40.734236 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.734202 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p\" (UniqueName: \"kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p\") pod \"22fcff7a-4655-438b-aa95-b4b95c0ce152\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " Oct 31 02:07:40.734437 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.734268 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca\") pod \"22fcff7a-4655-438b-aa95-b4b95c0ce152\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " Oct 31 02:07:40.734437 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.734328 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca\") pod \"22fcff7a-4655-438b-aa95-b4b95c0ce152\" (UID: \"22fcff7a-4655-438b-aa95-b4b95c0ce152\") " Oct 31 02:07:40.734648 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.734626 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca" pod "22fcff7a-4655-438b-aa95-b4b95c0ce152" (UID: "22fcff7a-4655-438b-aa95-b4b95c0ce152"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:40.734796 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.734783 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:40.736513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.736490 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca" pod "22fcff7a-4655-438b-aa95-b4b95c0ce152" (UID: "22fcff7a-4655-438b-aa95-b4b95c0ce152"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:07:40.736642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.736520 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p" pod "22fcff7a-4655-438b-aa95-b4b95c0ce152" (UID: "22fcff7a-4655-438b-aa95-b4b95c0ce152"). InnerVolumeSpecName "kube-api-access-mbm9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:40.836382 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.836346 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbm9p\" (UniqueName: \"kubernetes.io/projected/22fcff7a-4655-438b-aa95-b4b95c0ce152-kube-api-access-mbm9p\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:40.836382 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:40.836380 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fcff7a-4655-438b-aa95-b4b95c0ce152-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:07:41.521071 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:41.521038 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" Oct 31 02:07:41.521268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:41.521076 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw" event={"ID":"22fcff7a-4655-438b-aa95-b4b95c0ce152","Type":"ContainerDied","Data":"e6049dda7bd3687310422a31b6d32861a396014721db98012e7d2bba6e5bfc0a"} Oct 31 02:07:41.521268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:41.521130 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6049dda7bd3687310422a31b6d32861a396014721db98012e7d2bba6e5bfc0a" Oct 31 02:07:42.852989 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:42.852956 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:07:49.245297 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:49.245218 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd"] Oct 31 02:07:49.348994 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:49.348959 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f"] Oct 31 02:07:52.616046 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:52.615963 5756 generic.go:356] "Generic (PLEG): container finished" podID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerID="8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d" exitCode=0 Oct 31 02:07:52.616046 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:52.616023 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" event={"ID":"106248d9-94d9-4c0e-8dba-07fe9718d991","Type":"ContainerDied","Data":"8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d"} Oct 31 02:07:52.616625 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:52.616485 5756 scope.go:122] "RemoveContainer" containerID="8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d" Oct 31 02:07:52.617954 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:52.617927 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:07:52.618048 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:52.617993 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:07:52.618048 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:52.618024 5756 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 02:07:53.625769 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:53.625734 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" event={"ID":"106248d9-94d9-4c0e-8dba-07fe9718d991","Type":"ContainerStarted","Data":"f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5"} Oct 31 02:07:54.252920 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:54.248936 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerName="payload-processing" containerID="cri-o://5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4" gracePeriod=25 Oct 31 02:07:54.299184 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:54.299147 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:07:54.299392 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:54.299370 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 31 02:07:54.299485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:54.299439 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 31 02:07:54.353028 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:54.352977 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerName="payload-pre-processing" containerID="cri-o://0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688" gracePeriod=25 Oct 31 02:07:55.298634 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:55.298596 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 31 02:07:55.299048 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:55.298662 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 31 02:07:56.298393 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:56.298358 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 31 02:07:56.298582 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:56.298420 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 31 02:07:57.298705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:57.298672 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 31 02:07:57.299121 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:57.298725 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 31 02:07:58.298578 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:58.298547 5756 patch_prober.go:29] interesting pod/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" start-of-body= Oct 31 02:07:58.298759 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:58.298597 5756 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.254:15021/healthz/ready\": dial tcp 10.134.0.254:15021: connect: connection refused" Oct 31 02:07:58.379177 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:58.379137 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:07:58.379352 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:58.379179 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:07:59.302999 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:59.302970 5756 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:07:59.303491 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:59.303281 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:07:59.303949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:07:59.303932 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:08:02.066776 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:08:02.066733 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:08:08.378790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:08.378745 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:08:08.378790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:08.378774 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:08:09.430170 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.430135 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg"] Oct 31 02:08:09.430743 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.430728 5756 cpu_manager.go:414] "containerMap: removing container" podUID="22fcff7a-4655-438b-aa95-b4b95c0ce152" containerName="revoke-keys" Oct 31 02:08:09.430802 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.430745 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fcff7a-4655-438b-aa95-b4b95c0ce152" containerName="revoke-keys" Oct 31 02:08:09.430844 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.430826 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fcff7a-4655-438b-aa95-b4b95c0ce152" containerName="revoke-keys" Oct 31 02:08:09.433343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.433327 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.485012 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.484974 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgl7n\" (UniqueName: \"kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.485167 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.485123 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.485167 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.485146 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.486307 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.486284 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc"] Oct 31 02:08:09.489250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.489232 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.533092 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.533062 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:08:09.533432 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.533410 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:08:09.547806 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.547772 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg"] Oct 31 02:08:09.550410 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.550385 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc"] Oct 31 02:08:09.586001 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.585970 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86blp\" (UniqueName: \"kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.586197 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.586108 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.586197 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.586146 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.586328 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.586205 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.586328 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.586244 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.586446 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.586426 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgl7n\" (UniqueName: \"kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.586923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.586896 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.588620 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.588600 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.646593 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.646559 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgl7n\" (UniqueName: \"kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n\") pod \"maas-api-revoke-sub-1c422736-mf2fg\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.687379 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.687315 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86blp\" (UniqueName: \"kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.687525 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.687386 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.687525 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.687415 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.688066 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.688038 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.689831 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.689810 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.745033 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.745007 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:09.750571 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.750548 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86blp\" (UniqueName: \"kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp\") pod \"maas-api-revoke-sub-0966fd06-bhhsc\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:09.802259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:09.801875 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:10.128887 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.128838 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg"] Oct 31 02:08:10.130637 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:08:10.130606 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eff32c_e918_4a22_ade4_0ff28696fff4.slice/crio-44cceadcbd1860b5d4e5b5f98bd045ca85394bec30ac06da84f90fffc1df9b9b WatchSource:0}: Error finding container 44cceadcbd1860b5d4e5b5f98bd045ca85394bec30ac06da84f90fffc1df9b9b: Status 404 returned error can't find the container with id 44cceadcbd1860b5d4e5b5f98bd045ca85394bec30ac06da84f90fffc1df9b9b Oct 31 02:08:10.190645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.190617 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc"] Oct 31 02:08:10.191602 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:08:10.191570 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf347bbd_c700_41b4_b5c1_44f637bffab8.slice/crio-d9d9c0ee3249c254a8e9330bb9cb5df1d4a4b8a1ce497239d404722431f247b2 WatchSource:0}: Error finding container d9d9c0ee3249c254a8e9330bb9cb5df1d4a4b8a1ce497239d404722431f247b2: Status 404 returned error can't find the container with id d9d9c0ee3249c254a8e9330bb9cb5df1d4a4b8a1ce497239d404722431f247b2 Oct 31 02:08:10.767369 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.767268 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" event={"ID":"bf347bbd-c700-41b4-b5c1-44f637bffab8","Type":"ContainerStarted","Data":"dd43727f25daa06354c59075c0af0e002400f922b5e1a126bf2b5b0225e3915d"} Oct 31 02:08:10.767369 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.767338 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" event={"ID":"bf347bbd-c700-41b4-b5c1-44f637bffab8","Type":"ContainerStarted","Data":"d9d9c0ee3249c254a8e9330bb9cb5df1d4a4b8a1ce497239d404722431f247b2"} Oct 31 02:08:10.769073 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.769049 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" event={"ID":"c9eff32c-e918-4a22-ade4-0ff28696fff4","Type":"ContainerStarted","Data":"c8d1e72d5e5756afb1d3391c0d84079861d0eef29f73e88fd9598c24d42cfc8d"} Oct 31 02:08:10.769213 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.769090 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" event={"ID":"c9eff32c-e918-4a22-ade4-0ff28696fff4","Type":"ContainerStarted","Data":"44cceadcbd1860b5d4e5b5f98bd045ca85394bec30ac06da84f90fffc1df9b9b"} Oct 31 02:08:10.892388 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:10.892324 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" podStartSLOduration=1.892307921 podStartE2EDuration="1.892307921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:10.891121895 +0000 UTC m=+4262.698089438" watchObservedRunningTime="2026-10-31 02:08:10.892307921 +0000 UTC m=+4262.699275467" Oct 31 02:08:11.006776 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:11.006720 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" podStartSLOduration=2.006704425 podStartE2EDuration="2.006704425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:11.005530874 +0000 UTC m=+4262.812498420" watchObservedRunningTime="2026-10-31 02:08:11.006704425 +0000 UTC m=+4262.813671968" Oct 31 02:08:11.777268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:11.777171 5756 generic.go:356] "Generic (PLEG): container finished" podID="bf347bbd-c700-41b4-b5c1-44f637bffab8" containerID="dd43727f25daa06354c59075c0af0e002400f922b5e1a126bf2b5b0225e3915d" exitCode=0 Oct 31 02:08:11.777268 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:11.777249 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" event={"ID":"bf347bbd-c700-41b4-b5c1-44f637bffab8","Type":"ContainerDied","Data":"dd43727f25daa06354c59075c0af0e002400f922b5e1a126bf2b5b0225e3915d"} Oct 31 02:08:11.778956 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:11.778919 5756 generic.go:356] "Generic (PLEG): container finished" podID="c9eff32c-e918-4a22-ade4-0ff28696fff4" containerID="c8d1e72d5e5756afb1d3391c0d84079861d0eef29f73e88fd9598c24d42cfc8d" exitCode=0 Oct 31 02:08:11.779101 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:11.778998 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" event={"ID":"c9eff32c-e918-4a22-ade4-0ff28696fff4","Type":"ContainerDied","Data":"c8d1e72d5e5756afb1d3391c0d84079861d0eef29f73e88fd9598c24d42cfc8d"} Oct 31 02:08:12.938025 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:12.937972 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:12.973586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:12.973565 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:13.027900 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.027847 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca\") pod \"bf347bbd-c700-41b4-b5c1-44f637bffab8\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " Oct 31 02:08:13.028125 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.028017 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca\") pod \"bf347bbd-c700-41b4-b5c1-44f637bffab8\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " Oct 31 02:08:13.028125 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.028084 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp\" (UniqueName: \"kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp\") pod \"bf347bbd-c700-41b4-b5c1-44f637bffab8\" (UID: \"bf347bbd-c700-41b4-b5c1-44f637bffab8\") " Oct 31 02:08:13.028281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.028145 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca\") pod \"c9eff32c-e918-4a22-ade4-0ff28696fff4\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " Oct 31 02:08:13.028281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.028243 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n\" (UniqueName: \"kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n\") pod \"c9eff32c-e918-4a22-ade4-0ff28696fff4\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " Oct 31 02:08:13.028281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.028269 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca\") pod \"c9eff32c-e918-4a22-ade4-0ff28696fff4\" (UID: \"c9eff32c-e918-4a22-ade4-0ff28696fff4\") " Oct 31 02:08:13.030479 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.029755 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca" pod "bf347bbd-c700-41b4-b5c1-44f637bffab8" (UID: "bf347bbd-c700-41b4-b5c1-44f637bffab8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:13.030479 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.030438 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca" pod "c9eff32c-e918-4a22-ade4-0ff28696fff4" (UID: "c9eff32c-e918-4a22-ade4-0ff28696fff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:13.031475 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.031453 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp" pod "bf347bbd-c700-41b4-b5c1-44f637bffab8" (UID: "bf347bbd-c700-41b4-b5c1-44f637bffab8"). InnerVolumeSpecName "kube-api-access-86blp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:13.032071 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.032043 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca" pod "c9eff32c-e918-4a22-ade4-0ff28696fff4" (UID: "c9eff32c-e918-4a22-ade4-0ff28696fff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:08:13.032791 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.032767 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n" pod "c9eff32c-e918-4a22-ade4-0ff28696fff4" (UID: "c9eff32c-e918-4a22-ade4-0ff28696fff4"). InnerVolumeSpecName "kube-api-access-qgl7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:13.035222 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.035201 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca" pod "bf347bbd-c700-41b4-b5c1-44f637bffab8" (UID: "bf347bbd-c700-41b4-b5c1-44f637bffab8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:08:13.130191 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.130159 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:13.130191 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.130190 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86blp\" (UniqueName: \"kubernetes.io/projected/bf347bbd-c700-41b4-b5c1-44f637bffab8-kube-api-access-86blp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:13.130471 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.130201 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:13.130471 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.130210 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgl7n\" (UniqueName: \"kubernetes.io/projected/c9eff32c-e918-4a22-ade4-0ff28696fff4-kube-api-access-qgl7n\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:13.130471 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.130218 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9eff32c-e918-4a22-ade4-0ff28696fff4-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:13.130471 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.130227 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf347bbd-c700-41b4-b5c1-44f637bffab8-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:13.792646 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.792617 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" Oct 31 02:08:13.792646 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.792636 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc" event={"ID":"bf347bbd-c700-41b4-b5c1-44f637bffab8","Type":"ContainerDied","Data":"d9d9c0ee3249c254a8e9330bb9cb5df1d4a4b8a1ce497239d404722431f247b2"} Oct 31 02:08:13.792959 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.792689 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d9c0ee3249c254a8e9330bb9cb5df1d4a4b8a1ce497239d404722431f247b2" Oct 31 02:08:13.794679 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.794651 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" Oct 31 02:08:13.794829 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.794688 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg" event={"ID":"c9eff32c-e918-4a22-ade4-0ff28696fff4","Type":"ContainerDied","Data":"44cceadcbd1860b5d4e5b5f98bd045ca85394bec30ac06da84f90fffc1df9b9b"} Oct 31 02:08:13.794829 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:13.794732 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44cceadcbd1860b5d4e5b5f98bd045ca85394bec30ac06da84f90fffc1df9b9b" Oct 31 02:08:14.526659 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.526602 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 02:08:14.604765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.604734 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 02:08:14.645293 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.645263 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume\") pod \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " Oct 31 02:08:14.645458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.645336 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8\" (UniqueName: \"kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8\") pod \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " Oct 31 02:08:14.645458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.645387 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x\" (UniqueName: \"kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x\") pod \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\" (UID: \"afdcf97a-56f4-4b9e-a347-1296bc20c7bf\") " Oct 31 02:08:14.645539 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.645466 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume\") pod \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\" (UID: \"f73fc1fe-ef05-4cff-ad53-52ebf7588186\") " Oct 31 02:08:14.645846 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.645684 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume" pod "afdcf97a-56f4-4b9e-a347-1296bc20c7bf" (UID: "afdcf97a-56f4-4b9e-a347-1296bc20c7bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:14.646004 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.645887 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume" pod "f73fc1fe-ef05-4cff-ad53-52ebf7588186" (UID: "f73fc1fe-ef05-4cff-ad53-52ebf7588186"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:14.647542 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.647519 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8" pod "f73fc1fe-ef05-4cff-ad53-52ebf7588186" (UID: "f73fc1fe-ef05-4cff-ad53-52ebf7588186"). InnerVolumeSpecName "kube-api-access-psqs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:14.647629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.647611 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x" pod "afdcf97a-56f4-4b9e-a347-1296bc20c7bf" (UID: "afdcf97a-56f4-4b9e-a347-1296bc20c7bf"). InnerVolumeSpecName "kube-api-access-r298x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:14.746678 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.746645 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f73fc1fe-ef05-4cff-ad53-52ebf7588186-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:14.746678 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.746674 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:14.746678 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.746683 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psqs8\" (UniqueName: \"kubernetes.io/projected/f73fc1fe-ef05-4cff-ad53-52ebf7588186-kube-api-access-psqs8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:14.746952 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.746691 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r298x\" (UniqueName: \"kubernetes.io/projected/afdcf97a-56f4-4b9e-a347-1296bc20c7bf-kube-api-access-r298x\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:14.803685 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.803653 5756 generic.go:356] "Generic (PLEG): container finished" podID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerID="5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4" exitCode=0 Oct 31 02:08:14.803901 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.803701 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" Oct 31 02:08:14.803901 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.803743 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" event={"ID":"f73fc1fe-ef05-4cff-ad53-52ebf7588186","Type":"ContainerDied","Data":"5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4"} Oct 31 02:08:14.803901 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.803795 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd" event={"ID":"f73fc1fe-ef05-4cff-ad53-52ebf7588186","Type":"ContainerDied","Data":"2796aec83f7914ad347912b40700be6c1724281570f124561dbabdbaa8232560"} Oct 31 02:08:14.803901 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.803827 5756 scope.go:122] "RemoveContainer" containerID="5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4" Oct 31 02:08:14.805368 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.805343 5756 generic.go:356] "Generic (PLEG): container finished" podID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerID="0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688" exitCode=0 Oct 31 02:08:14.805642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.805371 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" event={"ID":"afdcf97a-56f4-4b9e-a347-1296bc20c7bf","Type":"ContainerDied","Data":"0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688"} Oct 31 02:08:14.805642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.805394 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" Oct 31 02:08:14.805642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.805414 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f" event={"ID":"afdcf97a-56f4-4b9e-a347-1296bc20c7bf","Type":"ContainerDied","Data":"95bdd0d84aa47e88e995d3cd99f605c038d55b9923226f4d41526a772c731a6d"} Oct 31 02:08:14.819318 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.819301 5756 scope.go:122] "RemoveContainer" containerID="5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4" Oct 31 02:08:14.821021 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:08:14.820988 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4\": container with ID starting with 5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4 not found: ID does not exist" containerID="5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4" Oct 31 02:08:14.821140 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.821031 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4"} err="failed to get container status \"5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4\": rpc error: code = NotFound desc = could not find container \"5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4\": container with ID starting with 5e16767e80c1440f6c157279a73c0ddda884d019ec2ff4a833e9eecc77e566c4 not found: ID does not exist" Oct 31 02:08:14.821140 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.821054 5756 scope.go:122] "RemoveContainer" containerID="0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688" Oct 31 02:08:14.832710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.832688 5756 scope.go:122] "RemoveContainer" containerID="0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688" Oct 31 02:08:14.833112 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:08:14.833072 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688\": container with ID starting with 0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688 not found: ID does not exist" containerID="0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688" Oct 31 02:08:14.833210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.833109 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688"} err="failed to get container status \"0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688\": rpc error: code = NotFound desc = could not find container \"0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688\": container with ID starting with 0dce26118f46eccb9caa51b4d80e78a6549097175c660d45c97110b674bd4688 not found: ID does not exist" Oct 31 02:08:14.969261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.969225 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd"] Oct 31 02:08:14.973385 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:14.973357 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-41014a-5758d9cf7c-r89bd"] Oct 31 02:08:15.081177 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:15.081087 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f"] Oct 31 02:08:15.084968 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:15.084944 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-41014a-8446845c47-2pw7f"] Oct 31 02:08:16.311393 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:16.311362 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" path="/var/lib/kubelet/pods/afdcf97a-56f4-4b9e-a347-1296bc20c7bf/volumes" Oct 31 02:08:16.311780 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:16.311750 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" path="/var/lib/kubelet/pods/f73fc1fe-ef05-4cff-ad53-52ebf7588186/volumes" Oct 31 02:08:21.221276 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:21.221242 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg"] Oct 31 02:08:21.232358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:21.232326 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bbe5e0-6jfhg"] Oct 31 02:08:22.311661 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:22.311629 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf0ed839-7e95-427c-b842-9397caf0d32c" path="/var/lib/kubelet/pods/bf0ed839-7e95-427c-b842-9397caf0d32c/volumes" Oct 31 02:08:27.866978 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:27.866947 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx"] Oct 31 02:08:27.867384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:27.867165 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" containerID="cri-o://93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9" gracePeriod=30 Oct 31 02:08:28.667300 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.667263 5756 patch_prober.go:29] interesting pod/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:08:28.667568 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.667331 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:08:28.869689 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.869661 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz"] Oct 31 02:08:28.870211 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870197 5756 cpu_manager.go:414] "containerMap: removing container" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerName="payload-pre-processing" Oct 31 02:08:28.870273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870212 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerName="payload-pre-processing" Oct 31 02:08:28.870273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870232 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c9eff32c-e918-4a22-ade4-0ff28696fff4" containerName="revoke-keys" Oct 31 02:08:28.870273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870238 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9eff32c-e918-4a22-ade4-0ff28696fff4" containerName="revoke-keys" Oct 31 02:08:28.870273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870255 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerName="payload-processing" Oct 31 02:08:28.870273 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870260 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerName="payload-processing" Oct 31 02:08:28.870439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870283 5756 cpu_manager.go:414] "containerMap: removing container" podUID="bf347bbd-c700-41b4-b5c1-44f637bffab8" containerName="revoke-keys" Oct 31 02:08:28.870439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870288 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf347bbd-c700-41b4-b5c1-44f637bffab8" containerName="revoke-keys" Oct 31 02:08:28.870439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870361 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf347bbd-c700-41b4-b5c1-44f637bffab8" containerName="revoke-keys" Oct 31 02:08:28.870439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870373 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="afdcf97a-56f4-4b9e-a347-1296bc20c7bf" containerName="payload-pre-processing" Oct 31 02:08:28.870439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870381 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9eff32c-e918-4a22-ade4-0ff28696fff4" containerName="revoke-keys" Oct 31 02:08:28.870439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.870388 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f73fc1fe-ef05-4cff-ad53-52ebf7588186" containerName="payload-processing" Oct 31 02:08:28.874917 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.874891 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:28.930230 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.930152 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz"] Oct 31 02:08:28.974949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.974917 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:08:28.975603 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:28.975583 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:08:29.014244 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.014211 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.014423 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.014348 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4swv\" (UniqueName: \"kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.014506 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.014441 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.115680 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.115644 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4swv\" (UniqueName: \"kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.115895 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.115712 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.115895 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.115779 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.116349 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.116326 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.118287 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.118264 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.173375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.173347 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4swv\" (UniqueName: \"kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.189246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.189187 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:29.576240 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.576213 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz"] Oct 31 02:08:29.578326 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:08:29.578300 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2546793b_d27e_4cfb_9873_6309acaac85a.slice/crio-7f6dec403e509181b70544b61b857736674e30f2d4d58731a51980cf72507b12 WatchSource:0}: Error finding container 7f6dec403e509181b70544b61b857736674e30f2d4d58731a51980cf72507b12: Status 404 returned error can't find the container with id 7f6dec403e509181b70544b61b857736674e30f2d4d58731a51980cf72507b12 Oct 31 02:08:29.902527 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.902488 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" event={"ID":"2546793b-d27e-4cfb-9873-6309acaac85a","Type":"ContainerStarted","Data":"6997e2eefd7f89ae779fa18aa529069fda49d947e7cc8b5a9a5b33b3e7d68c06"} Oct 31 02:08:29.902930 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:29.902566 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" event={"ID":"2546793b-d27e-4cfb-9873-6309acaac85a","Type":"ContainerStarted","Data":"7f6dec403e509181b70544b61b857736674e30f2d4d58731a51980cf72507b12"} Oct 31 02:08:30.016692 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:30.016636 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" podStartSLOduration=2.016620939 podStartE2EDuration="2.016620939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:30.015782183 +0000 UTC m=+4281.822749732" watchObservedRunningTime="2026-10-31 02:08:30.016620939 +0000 UTC m=+4281.823588482" Oct 31 02:08:30.910929 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:30.910895 5756 generic.go:356] "Generic (PLEG): container finished" podID="2546793b-d27e-4cfb-9873-6309acaac85a" containerID="6997e2eefd7f89ae779fa18aa529069fda49d947e7cc8b5a9a5b33b3e7d68c06" exitCode=0 Oct 31 02:08:30.911401 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:30.910946 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" event={"ID":"2546793b-d27e-4cfb-9873-6309acaac85a","Type":"ContainerDied","Data":"6997e2eefd7f89ae779fa18aa529069fda49d947e7cc8b5a9a5b33b3e7d68c06"} Oct 31 02:08:32.064205 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.064177 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:32.068151 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:08:32.068119 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:08:32.151684 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.151648 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv\" (UniqueName: \"kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv\") pod \"2546793b-d27e-4cfb-9873-6309acaac85a\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " Oct 31 02:08:32.151902 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.151702 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca\") pod \"2546793b-d27e-4cfb-9873-6309acaac85a\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " Oct 31 02:08:32.151902 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.151744 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca\") pod \"2546793b-d27e-4cfb-9873-6309acaac85a\" (UID: \"2546793b-d27e-4cfb-9873-6309acaac85a\") " Oct 31 02:08:32.152249 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.152216 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca" pod "2546793b-d27e-4cfb-9873-6309acaac85a" (UID: "2546793b-d27e-4cfb-9873-6309acaac85a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:32.153951 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.153928 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca" pod "2546793b-d27e-4cfb-9873-6309acaac85a" (UID: "2546793b-d27e-4cfb-9873-6309acaac85a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:08:32.154052 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.153998 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv" pod "2546793b-d27e-4cfb-9873-6309acaac85a" (UID: "2546793b-d27e-4cfb-9873-6309acaac85a"). InnerVolumeSpecName "kube-api-access-d4swv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:32.185607 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.185574 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:08:32.253790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.253711 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4swv\" (UniqueName: \"kubernetes.io/projected/2546793b-d27e-4cfb-9873-6309acaac85a-kube-api-access-d4swv\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:32.253790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.253740 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:32.253790 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.253749 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2546793b-d27e-4cfb-9873-6309acaac85a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:32.925306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.925269 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" Oct 31 02:08:32.925306 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.925299 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz" event={"ID":"2546793b-d27e-4cfb-9873-6309acaac85a","Type":"ContainerDied","Data":"7f6dec403e509181b70544b61b857736674e30f2d4d58731a51980cf72507b12"} Oct 31 02:08:32.925563 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:32.925347 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f6dec403e509181b70544b61b857736674e30f2d4d58731a51980cf72507b12" Oct 31 02:08:33.121304 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.121283 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:08:33.163617 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163542 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.163617 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163616 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.163811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163643 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.163811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163727 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.163811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163750 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.163972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163824 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:33.163972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163909 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.164068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.163970 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.164068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164006 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.164068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164055 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy\") pod \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\" (UID: \"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22\") " Oct 31 02:08:33.164241 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164074 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:33.164241 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164157 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:33.164241 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164184 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:33.164410 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164287 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:33.164760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164678 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.164760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164708 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.164760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164724 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.164760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164740 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.164760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.164753 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.166195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.166169 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "kube-api-access-jq622". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:33.166296 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.166176 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:33.166296 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.166264 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:08:33.166384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.166368 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy" pod "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" (UID: "1d9465b6-7e9b-47bc-81eb-f0fa2d778a22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:33.265959 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.265925 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.266118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.265964 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq622\" (UniqueName: \"kubernetes.io/projected/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-kube-api-access-jq622\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.266118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.265981 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.266118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.265995 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:33.935284 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.935246 5756 generic.go:356] "Generic (PLEG): container finished" podID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerID="93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9" exitCode=0 Oct 31 02:08:33.935469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.935293 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" event={"ID":"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22","Type":"ContainerDied","Data":"93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9"} Oct 31 02:08:33.935469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.935310 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" Oct 31 02:08:33.935469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.935362 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx" event={"ID":"1d9465b6-7e9b-47bc-81eb-f0fa2d778a22","Type":"ContainerDied","Data":"17e8a431b0de9bade8e8aaffdf42b258ada0ca57efc863b07032442879d89291"} Oct 31 02:08:33.935469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.935385 5756 scope.go:122] "RemoveContainer" containerID="93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9" Oct 31 02:08:33.948376 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.948358 5756 scope.go:122] "RemoveContainer" containerID="959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40" Oct 31 02:08:33.960281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.960258 5756 scope.go:122] "RemoveContainer" containerID="93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9" Oct 31 02:08:33.960569 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:08:33.960551 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9\": container with ID starting with 93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9 not found: ID does not exist" containerID="93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9" Oct 31 02:08:33.960625 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.960581 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9"} err="failed to get container status \"93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9\": rpc error: code = NotFound desc = could not find container \"93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9\": container with ID starting with 93d3bf7e000e7b07c67919056f22cae6f3f66e46516ee1bb5fc92d315b85f1a9 not found: ID does not exist" Oct 31 02:08:33.960625 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.960598 5756 scope.go:122] "RemoveContainer" containerID="959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40" Oct 31 02:08:33.960837 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:08:33.960825 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40\": container with ID starting with 959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40 not found: ID does not exist" containerID="959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40" Oct 31 02:08:33.960897 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:33.960843 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40"} err="failed to get container status \"959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40\": rpc error: code = NotFound desc = could not find container \"959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40\": container with ID starting with 959822b2d1fbbcf533fc0657de7181849522e5a283344c5770dbb27cb89c6e40 not found: ID does not exist" Oct 31 02:08:34.099272 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:34.099242 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx"] Oct 31 02:08:34.148792 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:34.148760 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bbe5e0-openshift-default-56656f665-6dbzx"] Oct 31 02:08:34.311297 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:34.311269 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" path="/var/lib/kubelet/pods/1d9465b6-7e9b-47bc-81eb-f0fa2d778a22/volumes" Oct 31 02:08:47.452712 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:47.452679 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr"] Oct 31 02:08:47.501564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:47.501526 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-caff19-zd2rr"] Oct 31 02:08:48.312423 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:48.312389 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f92b5f68-53bc-4f91-85ea-0677756c6066" path="/var/lib/kubelet/pods/f92b5f68-53bc-4f91-85ea-0677756c6066/volumes" Oct 31 02:08:54.205484 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.205449 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs"] Oct 31 02:08:54.206050 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.205731 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" containerID="cri-o://fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100" gracePeriod=30 Oct 31 02:08:54.842485 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.842454 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl"] Oct 31 02:08:54.843112 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843086 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" Oct 31 02:08:54.843112 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843107 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843134 5756 cpu_manager.go:414] "containerMap: removing container" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843140 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843159 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2546793b-d27e-4cfb-9873-6309acaac85a" containerName="revoke-keys" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843164 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2546793b-d27e-4cfb-9873-6309acaac85a" containerName="revoke-keys" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843262 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843274 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2546793b-d27e-4cfb-9873-6309acaac85a" containerName="revoke-keys" Oct 31 02:08:54.843325 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.843284 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9465b6-7e9b-47bc-81eb-f0fa2d778a22" containerName="istio-proxy" Oct 31 02:08:54.847135 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.847115 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:54.902768 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.902717 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b91499-dockercfg-857rv\"" Oct 31 02:08:54.902982 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.902892 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b91499\"" Oct 31 02:08:54.909451 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.909418 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl"] Oct 31 02:08:54.991187 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.991155 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7"] Oct 31 02:08:54.995300 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:54.995279 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.020579 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.020553 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.020739 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.020621 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gmp\" (UniqueName: \"kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp\") pod \"payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.055623 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.055593 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7"] Oct 31 02:08:55.121759 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.121678 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b91499-697b648765-58gq7\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.121943 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.121795 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.121943 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.121845 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrxc\" (UniqueName: \"kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc\") pod \"payload-processing-e2e-extmod-b-b91499-697b648765-58gq7\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.121943 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.121908 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gmp\" (UniqueName: \"kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp\") pod \"payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.122643 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.122617 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.184435 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.184402 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gmp\" (UniqueName: \"kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp\") pod \"payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.223259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.223229 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b91499-697b648765-58gq7\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.223646 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.223295 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrxc\" (UniqueName: \"kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc\") pod \"payload-processing-e2e-extmod-b-b91499-697b648765-58gq7\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.223946 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.223924 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b91499-697b648765-58gq7\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.273229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.273195 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h"] Oct 31 02:08:55.277579 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.277555 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.281725 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.281698 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrxc\" (UniqueName: \"kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc\") pod \"payload-processing-e2e-extmod-b-b91499-697b648765-58gq7\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.309656 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.309632 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:55.331740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.331711 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:08:55.331922 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.331852 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:08:55.334796 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.334726 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h"] Oct 31 02:08:55.424988 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.424954 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.425161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.425057 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gghp\" (UniqueName: \"kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.425161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.425093 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.460469 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.460438 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:55.526929 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.526258 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.527116 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.527095 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gghp\" (UniqueName: \"kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.527205 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.526914 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.527205 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.527138 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.529656 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.529630 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.584269 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.584236 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gghp\" (UniqueName: \"kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.603085 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.603058 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:55.694702 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.694668 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7"] Oct 31 02:08:55.701988 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:08:55.701962 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afc27c5_e168_4efb_9fb1_d77fed08594d.slice/crio-31fb00a1c19126147281f3260bacaaf5e6a83ef7c370f0ceb0aa37101c1fdd6c WatchSource:0}: Error finding container 31fb00a1c19126147281f3260bacaaf5e6a83ef7c370f0ceb0aa37101c1fdd6c: Status 404 returned error can't find the container with id 31fb00a1c19126147281f3260bacaaf5e6a83ef7c370f0ceb0aa37101c1fdd6c Oct 31 02:08:55.836814 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.836788 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl"] Oct 31 02:08:55.837841 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:08:55.837814 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0859dae0_de47_43e0_9aff_2d1caaae9760.slice/crio-04ef695b0ede6ba647ad6f30e5db7acfcbcf2d85270ce164ba363016a6ea2fe5 WatchSource:0}: Error finding container 04ef695b0ede6ba647ad6f30e5db7acfcbcf2d85270ce164ba363016a6ea2fe5: Status 404 returned error can't find the container with id 04ef695b0ede6ba647ad6f30e5db7acfcbcf2d85270ce164ba363016a6ea2fe5 Oct 31 02:08:55.978977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:55.978955 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h"] Oct 31 02:08:55.981001 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:08:55.980975 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ff5d81_d9b8_4cdd_b69d_7dfc4c192953.slice/crio-f2f91795a68b5ecb7a0ec3c760771e4cb469ca267a856122aaad795d9093ac8c WatchSource:0}: Error finding container f2f91795a68b5ecb7a0ec3c760771e4cb469ca267a856122aaad795d9093ac8c: Status 404 returned error can't find the container with id f2f91795a68b5ecb7a0ec3c760771e4cb469ca267a856122aaad795d9093ac8c Oct 31 02:08:56.082292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.082244 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" event={"ID":"6afc27c5-e168-4efb-9fb1-d77fed08594d","Type":"ContainerStarted","Data":"d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f"} Oct 31 02:08:56.082489 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.082319 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" event={"ID":"6afc27c5-e168-4efb-9fb1-d77fed08594d","Type":"ContainerStarted","Data":"31fb00a1c19126147281f3260bacaaf5e6a83ef7c370f0ceb0aa37101c1fdd6c"} Oct 31 02:08:56.082489 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.082376 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:08:56.083735 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.083705 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" event={"ID":"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953","Type":"ContainerStarted","Data":"f2f91795a68b5ecb7a0ec3c760771e4cb469ca267a856122aaad795d9093ac8c"} Oct 31 02:08:56.085211 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.085186 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" event={"ID":"0859dae0-de47-43e0-9aff-2d1caaae9760","Type":"ContainerStarted","Data":"d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209"} Oct 31 02:08:56.085330 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.085226 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" event={"ID":"0859dae0-de47-43e0-9aff-2d1caaae9760","Type":"ContainerStarted","Data":"04ef695b0ede6ba647ad6f30e5db7acfcbcf2d85270ce164ba363016a6ea2fe5"} Oct 31 02:08:56.085330 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.085260 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:08:56.196384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.196325 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" podStartSLOduration=2.196311797 podStartE2EDuration="2.196311797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:56.195931077 +0000 UTC m=+4308.002898621" watchObservedRunningTime="2026-10-31 02:08:56.196311797 +0000 UTC m=+4308.003279336" Oct 31 02:08:56.313388 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:56.313337 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" podStartSLOduration=2.313323552 podStartE2EDuration="2.313323552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:08:56.312574003 +0000 UTC m=+4308.119541547" watchObservedRunningTime="2026-10-31 02:08:56.313323552 +0000 UTC m=+4308.120291096" Oct 31 02:08:57.093611 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:57.093561 5756 generic.go:356] "Generic (PLEG): container finished" podID="14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" containerID="f253acaab7db8e0048cf75d58c79d8d9e29f30a4aa92e74e213398db4e4d173d" exitCode=0 Oct 31 02:08:57.093802 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:57.093621 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" event={"ID":"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953","Type":"ContainerDied","Data":"f253acaab7db8e0048cf75d58c79d8d9e29f30a4aa92e74e213398db4e4d173d"} Oct 31 02:08:58.267517 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.267485 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:58.360405 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.360305 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca\") pod \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " Oct 31 02:08:58.362656 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.362626 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca" pod "14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" (UID: "14ff5d81-d9b8-4cdd-b69d-7dfc4c192953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:08:58.461928 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.461888 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp\" (UniqueName: \"kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp\") pod \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " Oct 31 02:08:58.462134 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.461963 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca\") pod \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\" (UID: \"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953\") " Oct 31 02:08:58.462414 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.462378 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca" pod "14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" (UID: "14ff5d81-d9b8-4cdd-b69d-7dfc4c192953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:58.462562 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.462527 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:58.462562 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.462546 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:58.464742 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.464709 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp" pod "14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" (UID: "14ff5d81-d9b8-4cdd-b69d-7dfc4c192953"). InnerVolumeSpecName "kube-api-access-4gghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:58.564002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.563949 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gghp\" (UniqueName: \"kubernetes.io/projected/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953-kube-api-access-4gghp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:58.842773 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.842739 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw"] Oct 31 02:08:58.845977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:58.845954 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-41014a-7brsw"] Oct 31 02:08:59.110421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.110334 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" event={"ID":"14ff5d81-d9b8-4cdd-b69d-7dfc4c192953","Type":"ContainerDied","Data":"f2f91795a68b5ecb7a0ec3c760771e4cb469ca267a856122aaad795d9093ac8c"} Oct 31 02:08:59.110421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.110362 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h" Oct 31 02:08:59.110421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.110392 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2f91795a68b5ecb7a0ec3c760771e4cb469ca267a856122aaad795d9093ac8c" Oct 31 02:08:59.469275 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.469251 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:08:59.473719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.473694 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.473853 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.473736 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.473853 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.473762 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.473853 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.473801 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.474045 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.473894 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.474045 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.473962 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.474143 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474074 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:59.474143 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474087 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:59.474143 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474100 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.474143 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474111 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:08:59.474356 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474170 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.474356 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474206 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket\") pod \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\" (UID: \"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1\") " Oct 31 02:08:59.474458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474399 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:59.474757 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474730 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:59.475068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474907 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.475068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474930 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.475068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474946 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.475068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474960 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.475068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.474976 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.476204 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.476174 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:08:59.476204 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.476183 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:08:59.476465 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.476440 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:59.476566 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.476516 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4" pod "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" (UID: "2a8fdc46-dbf4-41eb-bdca-72d32e9938b1"). InnerVolumeSpecName "kube-api-access-4c6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:08:59.575787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.575754 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.575787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.575782 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c6p4\" (UniqueName: \"kubernetes.io/projected/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-kube-api-access-4c6p4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.575787 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.575795 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:08:59.576055 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:08:59.575805 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:00.120564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.120529 5756 generic.go:356] "Generic (PLEG): container finished" podID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerID="fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100" exitCode=0 Oct 31 02:09:00.120773 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.120586 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" event={"ID":"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1","Type":"ContainerDied","Data":"fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100"} Oct 31 02:09:00.120773 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.120634 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" event={"ID":"2a8fdc46-dbf4-41eb-bdca-72d32e9938b1","Type":"ContainerDied","Data":"90fee7b253de878449121db27edb6c115878810f7e1bf204d46dce1ddcc5f088"} Oct 31 02:09:00.120773 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.120662 5756 scope.go:122] "RemoveContainer" containerID="fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100" Oct 31 02:09:00.121002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.120825 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs" Oct 31 02:09:00.135276 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.135244 5756 scope.go:122] "RemoveContainer" containerID="0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471" Oct 31 02:09:00.148758 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.148729 5756 scope.go:122] "RemoveContainer" containerID="fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100" Oct 31 02:09:00.149149 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:00.149129 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100\": container with ID starting with fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100 not found: ID does not exist" containerID="fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100" Oct 31 02:09:00.149249 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.149159 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100"} err="failed to get container status \"fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100\": rpc error: code = NotFound desc = could not find container \"fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100\": container with ID starting with fa0eac6fdc14409d872dbcc88d52a912bb23242adfd8be586e425694610c0100 not found: ID does not exist" Oct 31 02:09:00.149249 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.149180 5756 scope.go:122] "RemoveContainer" containerID="0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471" Oct 31 02:09:00.149466 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:00.149450 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471\": container with ID starting with 0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471 not found: ID does not exist" containerID="0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471" Oct 31 02:09:00.149511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.149473 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471"} err="failed to get container status \"0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471\": rpc error: code = NotFound desc = could not find container \"0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471\": container with ID starting with 0c595fba62b7f668310970d1686e438a84836ffcd37f5a49e21cd7d789c6e471 not found: ID does not exist" Oct 31 02:09:00.285202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.285166 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs"] Oct 31 02:09:00.289400 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.289376 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-caff19-openshift-default-859f4f5fd9-lnwhs"] Oct 31 02:09:00.311553 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.311524 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fcff7a-4655-438b-aa95-b4b95c0ce152" path="/var/lib/kubelet/pods/22fcff7a-4655-438b-aa95-b4b95c0ce152/volumes" Oct 31 02:09:00.312022 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:00.312006 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" path="/var/lib/kubelet/pods/2a8fdc46-dbf4-41eb-bdca-72d32e9938b1/volumes" Oct 31 02:09:02.066088 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:02.066050 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:09:06.229416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.229382 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj"] Oct 31 02:09:06.336416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.336384 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d"] Oct 31 02:09:06.402129 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402090 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd"] Oct 31 02:09:06.402630 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402615 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" Oct 31 02:09:06.402705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402632 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" Oct 31 02:09:06.402705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402649 5756 cpu_manager.go:414] "containerMap: removing container" podUID="14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" containerName="revoke-keys" Oct 31 02:09:06.402705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402653 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" containerName="revoke-keys" Oct 31 02:09:06.402705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402685 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" Oct 31 02:09:06.402705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402690 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" Oct 31 02:09:06.402925 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402797 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" Oct 31 02:09:06.402925 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.402809 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" containerName="revoke-keys" Oct 31 02:09:06.406318 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.406302 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.455125 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.454991 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95x8j\" (UniqueName: \"kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.455385 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.455175 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.455385 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.455213 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.458676 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.458643 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:09:06.458840 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.458738 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:09:06.461510 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.461474 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd"] Oct 31 02:09:06.556445 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.556413 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.556627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.556456 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.556627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.556577 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95x8j\" (UniqueName: \"kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.557217 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.557193 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.558998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.558978 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.613442 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.613418 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95x8j\" (UniqueName: \"kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:06.719668 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:06.719635 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:07.097403 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:07.097373 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd"] Oct 31 02:09:07.097634 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:07.097614 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:09:07.097717 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:07.097689 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:09:07.100124 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:09:07.100085 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4772bf_fe92_4c7f_be5d_1bdce1ed732a.slice/crio-73fabf6a8b34ecb9a890e4b241660b8c768f8eb72ebe18c032e8e21059962b2b WatchSource:0}: Error finding container 73fabf6a8b34ecb9a890e4b241660b8c768f8eb72ebe18c032e8e21059962b2b: Status 404 returned error can't find the container with id 73fabf6a8b34ecb9a890e4b241660b8c768f8eb72ebe18c032e8e21059962b2b Oct 31 02:09:07.167379 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:07.167351 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" event={"ID":"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a","Type":"ContainerStarted","Data":"73fabf6a8b34ecb9a890e4b241660b8c768f8eb72ebe18c032e8e21059962b2b"} Oct 31 02:09:08.177564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:08.177534 5756 generic.go:356] "Generic (PLEG): container finished" podID="5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" containerID="6c1cbd2f06343b1d7b36c0bf6a2ff5c8df58dac4fda0d62937bd02f5f63e3411" exitCode=0 Oct 31 02:09:08.177971 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:08.177620 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" event={"ID":"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a","Type":"ContainerDied","Data":"6c1cbd2f06343b1d7b36c0bf6a2ff5c8df58dac4fda0d62937bd02f5f63e3411"} Oct 31 02:09:09.340201 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.340176 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:09.391892 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.391846 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca\") pod \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " Oct 31 02:09:09.392060 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.391969 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j\" (UniqueName: \"kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j\") pod \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " Oct 31 02:09:09.392060 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.392007 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca\") pod \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\" (UID: \"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a\") " Oct 31 02:09:09.392235 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.392212 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca" pod "5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" (UID: "5c4772bf-fe92-4c7f-be5d-1bdce1ed732a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:09.392391 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.392369 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:09.394147 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.394120 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j" pod "5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" (UID: "5c4772bf-fe92-4c7f-be5d-1bdce1ed732a"). InnerVolumeSpecName "kube-api-access-95x8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:09.394262 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.394193 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca" pod "5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" (UID: "5c4772bf-fe92-4c7f-be5d-1bdce1ed732a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:09:09.493344 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.493273 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95x8j\" (UniqueName: \"kubernetes.io/projected/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-kube-api-access-95x8j\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:09.493344 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:09.493301 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:10.200834 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:10.200796 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" event={"ID":"5c4772bf-fe92-4c7f-be5d-1bdce1ed732a","Type":"ContainerDied","Data":"73fabf6a8b34ecb9a890e4b241660b8c768f8eb72ebe18c032e8e21059962b2b"} Oct 31 02:09:10.201036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:10.200851 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73fabf6a8b34ecb9a890e4b241660b8c768f8eb72ebe18c032e8e21059962b2b" Oct 31 02:09:10.201036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:10.200813 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd" Oct 31 02:09:10.524496 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:10.524351 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz"] Oct 31 02:09:10.529709 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:10.529682 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b992fe-29rvz"] Oct 31 02:09:11.229732 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:11.229671 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" containerName="payload-processing" containerID="cri-o://91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7" gracePeriod=25 Oct 31 02:09:11.337573 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:11.337512 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerName="payload-pre-processing" containerID="cri-o://563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206" gracePeriod=25 Oct 31 02:09:12.311026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:12.310996 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2546793b-d27e-4cfb-9873-6309acaac85a" path="/var/lib/kubelet/pods/2546793b-d27e-4cfb-9873-6309acaac85a/volumes" Oct 31 02:09:12.693556 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:12.693467 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:09:12.693556 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:12.693507 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:09:21.748321 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:21.748234 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt"] Oct 31 02:09:21.757215 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:21.757187 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-26fbc4-vx7dt"] Oct 31 02:09:21.796380 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:21.796353 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:09:22.311847 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:22.311810 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9234a39f-dc39-49f2-9200-dbdc34a9bf98" path="/var/lib/kubelet/pods/9234a39f-dc39-49f2-9200-dbdc34a9bf98/volumes" Oct 31 02:09:22.693656 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:22.693561 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:09:22.693845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:22.693651 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:09:28.077423 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.077388 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv"] Oct 31 02:09:28.078041 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.078023 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" containerName="revoke-keys" Oct 31 02:09:28.078109 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.078042 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" containerName="revoke-keys" Oct 31 02:09:28.078148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.078143 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8fdc46-dbf4-41eb-bdca-72d32e9938b1" containerName="istio-proxy" Oct 31 02:09:28.078182 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.078157 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" containerName="revoke-keys" Oct 31 02:09:28.081762 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.081747 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.137584 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.137328 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv"] Oct 31 02:09:28.183645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.183614 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:09:28.183799 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.183703 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:09:28.201308 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.201279 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.201432 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.201393 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwtt2\" (UniqueName: \"kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.201478 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.201441 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.302619 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.302590 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.302804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.302667 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwtt2\" (UniqueName: \"kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.302804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.302705 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.303442 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.303422 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.305348 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.305327 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.359171 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.359106 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwtt2\" (UniqueName: \"kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.395018 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.394985 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:28.788398 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.788364 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv"] Oct 31 02:09:28.788599 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.788426 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg"] Oct 31 02:09:28.788599 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:28.788577 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" containerID="cri-o://6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1" gracePeriod=30 Oct 31 02:09:28.797896 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:09:28.797830 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e5e73c_9714_463e_aeda_f83d81acf42c.slice/crio-3a476f09675ca86e48811a007bc5529f1e78699c510cb3aaae311eb91c380e6b WatchSource:0}: Error finding container 3a476f09675ca86e48811a007bc5529f1e78699c510cb3aaae311eb91c380e6b: Status 404 returned error can't find the container with id 3a476f09675ca86e48811a007bc5529f1e78699c510cb3aaae311eb91c380e6b Oct 31 02:09:29.328483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:29.328401 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" event={"ID":"33e5e73c-9714-463e-aeda-f83d81acf42c","Type":"ContainerStarted","Data":"cac613e78449ca4fdecc9de246c3e3fb5cc513afaba93de7a44588d8d05ee561"} Oct 31 02:09:29.328483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:29.328459 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" event={"ID":"33e5e73c-9714-463e-aeda-f83d81acf42c","Type":"ContainerStarted","Data":"3a476f09675ca86e48811a007bc5529f1e78699c510cb3aaae311eb91c380e6b"} Oct 31 02:09:29.462614 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:29.462557 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" podStartSLOduration=1.462543645 podStartE2EDuration="1.462543645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:29.46123269 +0000 UTC m=+4341.268200232" watchObservedRunningTime="2026-10-31 02:09:29.462543645 +0000 UTC m=+4341.269511209" Oct 31 02:09:30.336683 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:30.336650 5756 generic.go:356] "Generic (PLEG): container finished" podID="33e5e73c-9714-463e-aeda-f83d81acf42c" containerID="cac613e78449ca4fdecc9de246c3e3fb5cc513afaba93de7a44588d8d05ee561" exitCode=0 Oct 31 02:09:30.337092 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:30.336734 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" event={"ID":"33e5e73c-9714-463e-aeda-f83d81acf42c","Type":"ContainerDied","Data":"cac613e78449ca4fdecc9de246c3e3fb5cc513afaba93de7a44588d8d05ee561"} Oct 31 02:09:31.508608 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.508587 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:31.532033 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.531991 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 02:09:31.539289 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539263 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2\" (UniqueName: \"kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2\") pod \"33e5e73c-9714-463e-aeda-f83d81acf42c\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " Oct 31 02:09:31.539439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539300 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm\" (UniqueName: \"kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm\") pod \"f1fc606c-1581-416c-ae52-67ead76bd713\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " Oct 31 02:09:31.539439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539335 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca\") pod \"33e5e73c-9714-463e-aeda-f83d81acf42c\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " Oct 31 02:09:31.539439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539420 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume\") pod \"f1fc606c-1581-416c-ae52-67ead76bd713\" (UID: \"f1fc606c-1581-416c-ae52-67ead76bd713\") " Oct 31 02:09:31.539612 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539449 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca\") pod \"33e5e73c-9714-463e-aeda-f83d81acf42c\" (UID: \"33e5e73c-9714-463e-aeda-f83d81acf42c\") " Oct 31 02:09:31.539910 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539883 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume" pod "f1fc606c-1581-416c-ae52-67ead76bd713" (UID: "f1fc606c-1581-416c-ae52-67ead76bd713"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:31.539910 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.539892 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca" pod "33e5e73c-9714-463e-aeda-f83d81acf42c" (UID: "33e5e73c-9714-463e-aeda-f83d81acf42c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:31.542133 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.542095 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2" pod "33e5e73c-9714-463e-aeda-f83d81acf42c" (UID: "33e5e73c-9714-463e-aeda-f83d81acf42c"). InnerVolumeSpecName "kube-api-access-hwtt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:31.543081 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.543053 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca" pod "33e5e73c-9714-463e-aeda-f83d81acf42c" (UID: "33e5e73c-9714-463e-aeda-f83d81acf42c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:09:31.546985 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.546958 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm" pod "f1fc606c-1581-416c-ae52-67ead76bd713" (UID: "f1fc606c-1581-416c-ae52-67ead76bd713"). InnerVolumeSpecName "kube-api-access-z65vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:31.593831 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.593806 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 02:09:31.640470 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.640436 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume\") pod \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " Oct 31 02:09:31.640661 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.640585 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958\" (UniqueName: \"kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958\") pod \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\" (UID: \"e9dec466-5822-41ca-ad30-4a8b3ba2b90b\") " Oct 31 02:09:31.640818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.640795 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume" pod "e9dec466-5822-41ca-ad30-4a8b3ba2b90b" (UID: "e9dec466-5822-41ca-ad30-4a8b3ba2b90b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:31.641018 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.641000 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fc606c-1581-416c-ae52-67ead76bd713-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:31.641086 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.641024 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:31.641086 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.641039 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwtt2\" (UniqueName: \"kubernetes.io/projected/33e5e73c-9714-463e-aeda-f83d81acf42c-kube-api-access-hwtt2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:31.641086 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.641052 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z65vm\" (UniqueName: \"kubernetes.io/projected/f1fc606c-1581-416c-ae52-67ead76bd713-kube-api-access-z65vm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:31.641086 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.641067 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e5e73c-9714-463e-aeda-f83d81acf42c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:31.641086 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.641081 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:31.642918 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.642893 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958" pod "e9dec466-5822-41ca-ad30-4a8b3ba2b90b" (UID: "e9dec466-5822-41ca-ad30-4a8b3ba2b90b"). InnerVolumeSpecName "kube-api-access-zc958". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:31.742099 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:31.742065 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc958\" (UniqueName: \"kubernetes.io/projected/e9dec466-5822-41ca-ad30-4a8b3ba2b90b-kube-api-access-zc958\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:32.067389 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:32.067355 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:09:32.355013 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.354981 5756 generic.go:356] "Generic (PLEG): container finished" podID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerID="563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206" exitCode=0 Oct 31 02:09:32.355173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.355024 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" Oct 31 02:09:32.355173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.355058 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" event={"ID":"e9dec466-5822-41ca-ad30-4a8b3ba2b90b","Type":"ContainerDied","Data":"563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206"} Oct 31 02:09:32.355173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.355111 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d" event={"ID":"e9dec466-5822-41ca-ad30-4a8b3ba2b90b","Type":"ContainerDied","Data":"4abcd13bc0e2084d18af1be91209c90126fbfa34cfe753c33ecdb92248840943"} Oct 31 02:09:32.355173 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.355141 5756 scope.go:122] "RemoveContainer" containerID="563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206" Oct 31 02:09:32.356646 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.356621 5756 generic.go:356] "Generic (PLEG): container finished" podID="f1fc606c-1581-416c-ae52-67ead76bd713" containerID="91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7" exitCode=0 Oct 31 02:09:32.356758 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.356678 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" Oct 31 02:09:32.356820 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.356676 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" event={"ID":"f1fc606c-1581-416c-ae52-67ead76bd713","Type":"ContainerDied","Data":"91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7"} Oct 31 02:09:32.356820 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.356810 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj" event={"ID":"f1fc606c-1581-416c-ae52-67ead76bd713","Type":"ContainerDied","Data":"4d635c6e888def66f5ca2e0dd3f2f70720662c959ad981e61eee5b47af72cbe9"} Oct 31 02:09:32.358449 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.358428 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" event={"ID":"33e5e73c-9714-463e-aeda-f83d81acf42c","Type":"ContainerDied","Data":"3a476f09675ca86e48811a007bc5529f1e78699c510cb3aaae311eb91c380e6b"} Oct 31 02:09:32.358449 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.358447 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv" Oct 31 02:09:32.358628 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.358470 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a476f09675ca86e48811a007bc5529f1e78699c510cb3aaae311eb91c380e6b" Oct 31 02:09:32.366358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.366332 5756 scope.go:122] "RemoveContainer" containerID="563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206" Oct 31 02:09:32.366653 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:32.366633 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206\": container with ID starting with 563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206 not found: ID does not exist" containerID="563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206" Oct 31 02:09:32.366731 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.366660 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206"} err="failed to get container status \"563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206\": rpc error: code = NotFound desc = could not find container \"563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206\": container with ID starting with 563448d9ede4515fe18c4163e52407927f1f2f54b76b7120f04bdad0f78ea206 not found: ID does not exist" Oct 31 02:09:32.366731 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.366676 5756 scope.go:122] "RemoveContainer" containerID="91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7" Oct 31 02:09:32.375574 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.375554 5756 scope.go:122] "RemoveContainer" containerID="91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7" Oct 31 02:09:32.375834 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:32.375808 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7\": container with ID starting with 91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7 not found: ID does not exist" containerID="91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7" Oct 31 02:09:32.375971 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.375840 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7"} err="failed to get container status \"91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7\": rpc error: code = NotFound desc = could not find container \"91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7\": container with ID starting with 91d961c5689bc3b55c596255862e0d870c67b0a5dde3e06703e54c7919d9fbe7 not found: ID does not exist" Oct 31 02:09:32.571895 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.571791 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d"] Oct 31 02:09:32.620315 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.620284 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-dc9e63-5b787b699f-hbf2d"] Oct 31 02:09:32.685846 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.685817 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj"] Oct 31 02:09:32.690137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.690112 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-dc9e63-6684fb4989-7g4gj"] Oct 31 02:09:32.922307 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.922240 5756 patch_prober.go:29] interesting pod/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:09:32.922307 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:32.922289 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:09:34.063958 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.063901 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 02:09:34.165193 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165160 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165372 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165239 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165372 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165294 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165372 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165337 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165523 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165410 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165523 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165458 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165523 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165468 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:09:34.165523 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165486 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165630 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165661 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:09:34.165748 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165682 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:09:34.165926 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165743 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket\") pod \"ec648a48-3742-4497-8e93-a52e12d689b0\" (UID: \"ec648a48-3742-4497-8e93-a52e12d689b0\") " Oct 31 02:09:34.165926 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165878 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:34.166028 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.165982 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:09:34.166233 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.166214 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec648a48-3742-4497-8e93-a52e12d689b0-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.166328 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.166239 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.166328 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.166268 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.166328 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.166282 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.166328 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.166295 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.167821 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.167801 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:09:34.167922 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.167893 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:34.168312 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.168295 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:09:34.168366 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.168331 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz" pod "ec648a48-3742-4497-8e93-a52e12d689b0" (UID: "ec648a48-3742-4497-8e93-a52e12d689b0"). InnerVolumeSpecName "kube-api-access-bxvtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:34.267585 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.267551 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec648a48-3742-4497-8e93-a52e12d689b0-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.267585 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.267579 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvtz\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-kube-api-access-bxvtz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.267585 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.267589 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec648a48-3742-4497-8e93-a52e12d689b0-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.267818 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.267597 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec648a48-3742-4497-8e93-a52e12d689b0-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:34.311406 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.311372 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" path="/var/lib/kubelet/pods/e9dec466-5822-41ca-ad30-4a8b3ba2b90b/volumes" Oct 31 02:09:34.311811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.311796 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" path="/var/lib/kubelet/pods/f1fc606c-1581-416c-ae52-67ead76bd713/volumes" Oct 31 02:09:34.373578 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.373552 5756 generic.go:356] "Generic (PLEG): container finished" podID="ec648a48-3742-4497-8e93-a52e12d689b0" containerID="6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1" exitCode=0 Oct 31 02:09:34.373740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.373598 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" Oct 31 02:09:34.373740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.373639 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" event={"ID":"ec648a48-3742-4497-8e93-a52e12d689b0","Type":"ContainerDied","Data":"6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1"} Oct 31 02:09:34.373740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.373693 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg" event={"ID":"ec648a48-3742-4497-8e93-a52e12d689b0","Type":"ContainerDied","Data":"e8c7400f815c7f2c47e90607f6eecc099623a535914110a9289edff15e2ccbee"} Oct 31 02:09:34.373740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.373721 5756 scope.go:122] "RemoveContainer" containerID="6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1" Oct 31 02:09:34.391698 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.391678 5756 scope.go:122] "RemoveContainer" containerID="6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1" Oct 31 02:09:34.392052 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:09:34.392030 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1\": container with ID starting with 6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1 not found: ID does not exist" containerID="6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1" Oct 31 02:09:34.392116 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.392060 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1"} err="failed to get container status \"6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1\": rpc error: code = NotFound desc = could not find container \"6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1\": container with ID starting with 6e40604c1c0c431158d02e3c1538eec0612c6694ccee4f965cdd43a2d59f13d1 not found: ID does not exist" Oct 31 02:09:34.536633 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.536591 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg"] Oct 31 02:09:34.540246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:34.540222 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-26fbc4-openshift-default-7cf9949555-mlblg"] Oct 31 02:09:36.311689 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:36.311653 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" path="/var/lib/kubelet/pods/ec648a48-3742-4497-8e93-a52e12d689b0/volumes" Oct 31 02:09:38.015484 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:38.015448 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h"] Oct 31 02:09:38.064609 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:38.064577 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-dc9e63-gv99h"] Oct 31 02:09:38.310925 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:38.310899 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ff5d81-d9b8-4cdd-b69d-7dfc4c192953" path="/var/lib/kubelet/pods/14ff5d81-d9b8-4cdd-b69d-7dfc4c192953/volumes" Oct 31 02:09:39.692230 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:39.692194 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6"] Oct 31 02:09:44.693191 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:44.693146 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" podUID="18978975-2d04-4264-859a-d9288ae28e25" containerName="payload-pre-processing" containerID="cri-o://bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f" gracePeriod=25 Oct 31 02:09:46.173402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173365 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz"] Oct 31 02:09:46.173940 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173923 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerName="payload-pre-processing" Oct 31 02:09:46.174002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173941 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerName="payload-pre-processing" Oct 31 02:09:46.174002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173967 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" containerName="payload-processing" Oct 31 02:09:46.174002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173972 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" containerName="payload-processing" Oct 31 02:09:46.174002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173991 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" Oct 31 02:09:46.174002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.173996 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" Oct 31 02:09:46.174002 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.174003 5756 cpu_manager.go:414] "containerMap: removing container" podUID="33e5e73c-9714-463e-aeda-f83d81acf42c" containerName="revoke-keys" Oct 31 02:09:46.174195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.174008 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e5e73c-9714-463e-aeda-f83d81acf42c" containerName="revoke-keys" Oct 31 02:09:46.174195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.174093 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec648a48-3742-4497-8e93-a52e12d689b0" containerName="istio-proxy" Oct 31 02:09:46.174195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.174103 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e5e73c-9714-463e-aeda-f83d81acf42c" containerName="revoke-keys" Oct 31 02:09:46.174195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.174120 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9dec466-5822-41ca-ad30-4a8b3ba2b90b" containerName="payload-pre-processing" Oct 31 02:09:46.174195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.174128 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fc606c-1581-416c-ae52-67ead76bd713" containerName="payload-processing" Oct 31 02:09:46.177200 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.177180 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.230072 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.230048 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:09:46.234084 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.234061 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:09:46.281816 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.281781 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz"] Oct 31 02:09:46.299200 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.299167 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.299345 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.299322 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.299389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.299375 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6h2p\" (UniqueName: \"kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.400294 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.400262 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.400489 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.400322 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6h2p\" (UniqueName: \"kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.400489 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.400380 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.401143 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.401118 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.402849 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.402829 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.460240 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.460170 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6h2p\" (UniqueName: \"kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.490820 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.490792 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:46.873155 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:46.871057 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz"] Oct 31 02:09:46.884199 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:09:46.884169 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870f1da9_b28c_4707_ad0e_790973d102a0.slice/crio-1136bd20208f1bff803473389b81d1a3c290b16b1b98e37d50b45c5ceaa4e7e4 WatchSource:0}: Error finding container 1136bd20208f1bff803473389b81d1a3c290b16b1b98e37d50b45c5ceaa4e7e4: Status 404 returned error can't find the container with id 1136bd20208f1bff803473389b81d1a3c290b16b1b98e37d50b45c5ceaa4e7e4 Oct 31 02:09:47.460349 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:47.460253 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" event={"ID":"870f1da9-b28c-4707-ad0e-790973d102a0","Type":"ContainerStarted","Data":"0c744f117ab345f6a4104642bdee926db656d702c80536d748df1192ad866e99"} Oct 31 02:09:47.460349 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:47.460326 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" event={"ID":"870f1da9-b28c-4707-ad0e-790973d102a0","Type":"ContainerStarted","Data":"1136bd20208f1bff803473389b81d1a3c290b16b1b98e37d50b45c5ceaa4e7e4"} Oct 31 02:09:48.469671 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:48.469639 5756 generic.go:356] "Generic (PLEG): container finished" podID="870f1da9-b28c-4707-ad0e-790973d102a0" containerID="0c744f117ab345f6a4104642bdee926db656d702c80536d748df1192ad866e99" exitCode=0 Oct 31 02:09:48.470116 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:48.469712 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" event={"ID":"870f1da9-b28c-4707-ad0e-790973d102a0","Type":"ContainerDied","Data":"0c744f117ab345f6a4104642bdee926db656d702c80536d748df1192ad866e99"} Oct 31 02:09:49.625079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.625055 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:49.738742 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.738167 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p\" (UniqueName: \"kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p\") pod \"870f1da9-b28c-4707-ad0e-790973d102a0\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " Oct 31 02:09:49.738742 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.738222 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca\") pod \"870f1da9-b28c-4707-ad0e-790973d102a0\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " Oct 31 02:09:49.738742 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.738253 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca\") pod \"870f1da9-b28c-4707-ad0e-790973d102a0\" (UID: \"870f1da9-b28c-4707-ad0e-790973d102a0\") " Oct 31 02:09:49.739163 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.739114 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca" pod "870f1da9-b28c-4707-ad0e-790973d102a0" (UID: "870f1da9-b28c-4707-ad0e-790973d102a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:49.741205 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.741165 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca" pod "870f1da9-b28c-4707-ad0e-790973d102a0" (UID: "870f1da9-b28c-4707-ad0e-790973d102a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:09:49.741329 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.741288 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p" pod "870f1da9-b28c-4707-ad0e-790973d102a0" (UID: "870f1da9-b28c-4707-ad0e-790973d102a0"). InnerVolumeSpecName "kube-api-access-d6h2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:49.839452 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.839425 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6h2p\" (UniqueName: \"kubernetes.io/projected/870f1da9-b28c-4707-ad0e-790973d102a0-kube-api-access-d6h2p\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:49.839452 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.839449 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:49.839452 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:49.839459 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/870f1da9-b28c-4707-ad0e-790973d102a0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:09:50.483680 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:50.483651 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" Oct 31 02:09:50.483854 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:50.483690 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz" event={"ID":"870f1da9-b28c-4707-ad0e-790973d102a0","Type":"ContainerDied","Data":"1136bd20208f1bff803473389b81d1a3c290b16b1b98e37d50b45c5ceaa4e7e4"} Oct 31 02:09:50.483854 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:50.483735 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1136bd20208f1bff803473389b81d1a3c290b16b1b98e37d50b45c5ceaa4e7e4" Oct 31 02:09:54.325878 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:54.325833 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" podUID="18978975-2d04-4264-859a-d9288ae28e25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:09:57.433389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:09:57.433361 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76"] Oct 31 02:10:02.066359 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:10:02.066322 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:10:02.433959 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:02.433857 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerName="payload-processing" containerID="cri-o://9ab2e62451c8245aa4a75c6b8d2bf452abc10e31acd9c539232d898169c4bf59" gracePeriod=25 Oct 31 02:10:02.866554 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:02.866519 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz"] Oct 31 02:10:02.916832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:02.916796 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-9c9c3c-zd9hz"] Oct 31 02:10:04.181663 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:04.181625 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:10:04.312733 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:04.312693 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870f1da9-b28c-4707-ad0e-790973d102a0" path="/var/lib/kubelet/pods/870f1da9-b28c-4707-ad0e-790973d102a0/volumes" Oct 31 02:10:04.325957 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:04.325902 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" podUID="18978975-2d04-4264-859a-d9288ae28e25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:10:04.965041 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:04.964982 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 02:10:05.104795 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.104757 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume\") pod \"18978975-2d04-4264-859a-d9288ae28e25\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " Oct 31 02:10:05.105006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.104837 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w\" (UniqueName: \"kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w\") pod \"18978975-2d04-4264-859a-d9288ae28e25\" (UID: \"18978975-2d04-4264-859a-d9288ae28e25\") " Oct 31 02:10:05.105465 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.105439 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume" pod "18978975-2d04-4264-859a-d9288ae28e25" (UID: "18978975-2d04-4264-859a-d9288ae28e25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:05.107482 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.107452 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w" pod "18978975-2d04-4264-859a-d9288ae28e25" (UID: "18978975-2d04-4264-859a-d9288ae28e25"). InnerVolumeSpecName "kube-api-access-6nq9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:05.206648 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.206611 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18978975-2d04-4264-859a-d9288ae28e25-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:05.206648 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.206642 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nq9w\" (UniqueName: \"kubernetes.io/projected/18978975-2d04-4264-859a-d9288ae28e25-kube-api-access-6nq9w\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:05.310370 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.310330 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd"] Oct 31 02:10:05.312951 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.312925 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-083e17-gbgbd"] Oct 31 02:10:05.580337 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.580249 5756 generic.go:356] "Generic (PLEG): container finished" podID="18978975-2d04-4264-859a-d9288ae28e25" containerID="bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f" exitCode=0 Oct 31 02:10:05.580337 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.580301 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" Oct 31 02:10:05.580541 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.580335 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" event={"ID":"18978975-2d04-4264-859a-d9288ae28e25","Type":"ContainerDied","Data":"bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f"} Oct 31 02:10:05.580541 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.580385 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6" event={"ID":"18978975-2d04-4264-859a-d9288ae28e25","Type":"ContainerDied","Data":"5852fe8700ad9b00776c8fb2250a56b21ae37d26e93b1ae6e7b0ac1050a7cbdc"} Oct 31 02:10:05.580541 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.580407 5756 scope.go:122] "RemoveContainer" containerID="bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f" Oct 31 02:10:05.590630 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.590608 5756 scope.go:122] "RemoveContainer" containerID="bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f" Oct 31 02:10:05.590930 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:10:05.590912 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f\": container with ID starting with bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f not found: ID does not exist" containerID="bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f" Oct 31 02:10:05.590998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.590941 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f"} err="failed to get container status \"bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f\": rpc error: code = NotFound desc = could not find container \"bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f\": container with ID starting with bc4184a5ab6cb50af03fc5716fd3677df0926344d333631688faf63c46bcfe5f not found: ID does not exist" Oct 31 02:10:05.743115 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.743084 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6"] Oct 31 02:10:05.747251 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:05.747228 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fc058-86886d6d99-8lmh6"] Oct 31 02:10:06.312637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:06.312593 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18978975-2d04-4264-859a-d9288ae28e25" path="/var/lib/kubelet/pods/18978975-2d04-4264-859a-d9288ae28e25/volumes" Oct 31 02:10:06.313349 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:06.313318 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4772bf-fe92-4c7f-be5d-1bdce1ed732a" path="/var/lib/kubelet/pods/5c4772bf-fe92-4c7f-be5d-1bdce1ed732a/volumes" Oct 31 02:10:08.498936 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:08.498893 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:10:10.861665 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.861635 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5"] Oct 31 02:10:10.862131 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.862117 5756 cpu_manager.go:414] "containerMap: removing container" podUID="18978975-2d04-4264-859a-d9288ae28e25" containerName="payload-pre-processing" Oct 31 02:10:10.862178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.862131 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18978975-2d04-4264-859a-d9288ae28e25" containerName="payload-pre-processing" Oct 31 02:10:10.862178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.862159 5756 cpu_manager.go:414] "containerMap: removing container" podUID="870f1da9-b28c-4707-ad0e-790973d102a0" containerName="revoke-keys" Oct 31 02:10:10.862178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.862165 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870f1da9-b28c-4707-ad0e-790973d102a0" containerName="revoke-keys" Oct 31 02:10:10.862283 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.862254 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="18978975-2d04-4264-859a-d9288ae28e25" containerName="payload-pre-processing" Oct 31 02:10:10.862283 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.862266 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="870f1da9-b28c-4707-ad0e-790973d102a0" containerName="revoke-keys" Oct 31 02:10:10.867107 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.867091 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:10.922364 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.922331 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5"] Oct 31 02:10:10.968628 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.968594 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:10.968801 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.968656 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:10.968801 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.968746 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksx5\" (UniqueName: \"kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:10.968935 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.968803 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:10:10.968935 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:10.968848 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:10:11.070407 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.070377 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksx5\" (UniqueName: \"kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.070580 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.070496 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.070580 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.070542 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.071335 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.071270 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.073139 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.073116 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.127604 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.127535 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksx5\" (UniqueName: \"kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5\") pod \"maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.181079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.181053 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:11.561409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.561369 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5"] Oct 31 02:10:11.564253 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:11.564225 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184c0f44_1cf3_4d6b_a263_2d6c25abc7b8.slice/crio-2362a98c2cc638f649d4518ffc8cc7603a258df91484523b44a837a8299f48d9 WatchSource:0}: Error finding container 2362a98c2cc638f649d4518ffc8cc7603a258df91484523b44a837a8299f48d9: Status 404 returned error can't find the container with id 2362a98c2cc638f649d4518ffc8cc7603a258df91484523b44a837a8299f48d9 Oct 31 02:10:11.623420 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:11.623391 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" event={"ID":"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8","Type":"ContainerStarted","Data":"2362a98c2cc638f649d4518ffc8cc7603a258df91484523b44a837a8299f48d9"} Oct 31 02:10:12.631405 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:12.631367 5756 generic.go:356] "Generic (PLEG): container finished" podID="184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" containerID="c680ce66ee7c53f10ef3b8a9e29c60baa71e2e883e5e4948bdb417320bf51640" exitCode=0 Oct 31 02:10:12.631799 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:12.631446 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" event={"ID":"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8","Type":"ContainerDied","Data":"c680ce66ee7c53f10ef3b8a9e29c60baa71e2e883e5e4948bdb417320bf51640"} Oct 31 02:10:13.393719 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.393683 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7"] Oct 31 02:10:13.397623 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.397602 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.453367 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.453335 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7"] Oct 31 02:10:13.495089 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.495056 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmj4\" (UniqueName: \"kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.495267 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.495188 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.495267 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.495214 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.596229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.596194 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.596229 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.596228 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.596496 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.596352 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmj4\" (UniqueName: \"kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.596992 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.596965 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.598692 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.598670 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.652628 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.652564 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmj4\" (UniqueName: \"kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4\") pod \"maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.710852 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.710821 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:13.786303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.786278 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:13.902907 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.902808 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca\") pod \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " Oct 31 02:10:13.903085 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.902946 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5\" (UniqueName: \"kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5\") pod \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " Oct 31 02:10:13.903085 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.902988 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca\") pod \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\" (UID: \"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8\") " Oct 31 02:10:13.903457 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.903426 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca" pod "184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" (UID: "184c0f44-1cf3-4d6b-a263-2d6c25abc7b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:13.905825 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.905727 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca" pod "184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" (UID: "184c0f44-1cf3-4d6b-a263-2d6c25abc7b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:13.905987 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:13.905909 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5" pod "184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" (UID: "184c0f44-1cf3-4d6b-a263-2d6c25abc7b8"). InnerVolumeSpecName "kube-api-access-sksx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:14.004063 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.004022 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:14.004063 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.004063 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sksx5\" (UniqueName: \"kubernetes.io/projected/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-kube-api-access-sksx5\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:14.004272 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.004079 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:14.092874 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.092835 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7"] Oct 31 02:10:14.094618 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:14.094582 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d950f0d_a992_4c37_8ad3_08fa95bd91f9.slice/crio-e03f4945c5ccb9d6341dec5e7c1604d1b923e2541973dd724800565cf3da5d0d WatchSource:0}: Error finding container e03f4945c5ccb9d6341dec5e7c1604d1b923e2541973dd724800565cf3da5d0d: Status 404 returned error can't find the container with id e03f4945c5ccb9d6341dec5e7c1604d1b923e2541973dd724800565cf3da5d0d Oct 31 02:10:14.645354 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.645316 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" event={"ID":"5d950f0d-a992-4c37-8ad3-08fa95bd91f9","Type":"ContainerStarted","Data":"d83ff3773d7cc8ee78788ea5d7e3d9c3ffd0406c88aca55dedd1bb4e54baaf9b"} Oct 31 02:10:14.645549 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.645375 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" event={"ID":"5d950f0d-a992-4c37-8ad3-08fa95bd91f9","Type":"ContainerStarted","Data":"e03f4945c5ccb9d6341dec5e7c1604d1b923e2541973dd724800565cf3da5d0d"} Oct 31 02:10:14.646680 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.646656 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" Oct 31 02:10:14.646813 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.646656 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5" event={"ID":"184c0f44-1cf3-4d6b-a263-2d6c25abc7b8","Type":"ContainerDied","Data":"2362a98c2cc638f649d4518ffc8cc7603a258df91484523b44a837a8299f48d9"} Oct 31 02:10:14.646813 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.646745 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2362a98c2cc638f649d4518ffc8cc7603a258df91484523b44a837a8299f48d9" Oct 31 02:10:14.760740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:14.760688 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" podStartSLOduration=1.760673063 podStartE2EDuration="1.760673063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:14.75931003 +0000 UTC m=+4386.566277573" watchObservedRunningTime="2026-10-31 02:10:14.760673063 +0000 UTC m=+4386.567640605" Oct 31 02:10:15.328058 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:15.328026 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj"] Oct 31 02:10:15.341441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:15.341409 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8"] Oct 31 02:10:15.377069 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:15.377036 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7"] Oct 31 02:10:15.379038 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:15.379013 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n"] Oct 31 02:10:15.654987 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:15.654904 5756 generic.go:356] "Generic (PLEG): container finished" podID="5d950f0d-a992-4c37-8ad3-08fa95bd91f9" containerID="d83ff3773d7cc8ee78788ea5d7e3d9c3ffd0406c88aca55dedd1bb4e54baaf9b" exitCode=0 Oct 31 02:10:15.655126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:15.655037 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" event={"ID":"5d950f0d-a992-4c37-8ad3-08fa95bd91f9","Type":"ContainerDied","Data":"d83ff3773d7cc8ee78788ea5d7e3d9c3ffd0406c88aca55dedd1bb4e54baaf9b"} Oct 31 02:10:16.807463 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.807435 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:16.910639 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.910603 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv"] Oct 31 02:10:16.912498 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.912450 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7fc058-9kkzv"] Oct 31 02:10:16.935780 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.935754 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca\") pod \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " Oct 31 02:10:16.935972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.935809 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca\") pod \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " Oct 31 02:10:16.935972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.935922 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4\" (UniqueName: \"kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4\") pod \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\" (UID: \"5d950f0d-a992-4c37-8ad3-08fa95bd91f9\") " Oct 31 02:10:16.936208 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.936180 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca" pod "5d950f0d-a992-4c37-8ad3-08fa95bd91f9" (UID: "5d950f0d-a992-4c37-8ad3-08fa95bd91f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:16.936361 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.936344 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:16.938103 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.938076 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca" pod "5d950f0d-a992-4c37-8ad3-08fa95bd91f9" (UID: "5d950f0d-a992-4c37-8ad3-08fa95bd91f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:16.938202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:16.938156 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4" pod "5d950f0d-a992-4c37-8ad3-08fa95bd91f9" (UID: "5d950f0d-a992-4c37-8ad3-08fa95bd91f9"). InnerVolumeSpecName "kube-api-access-dhmj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:17.037979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:17.037947 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:17.037979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:17.037981 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmj4\" (UniqueName: \"kubernetes.io/projected/5d950f0d-a992-4c37-8ad3-08fa95bd91f9-kube-api-access-dhmj4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:17.675321 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:17.675281 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" event={"ID":"5d950f0d-a992-4c37-8ad3-08fa95bd91f9","Type":"ContainerDied","Data":"e03f4945c5ccb9d6341dec5e7c1604d1b923e2541973dd724800565cf3da5d0d"} Oct 31 02:10:17.675537 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:17.675343 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03f4945c5ccb9d6341dec5e7c1604d1b923e2541973dd724800565cf3da5d0d" Oct 31 02:10:17.675537 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:17.675283 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7" Oct 31 02:10:18.311500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:18.311472 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e5e73c-9714-463e-aeda-f83d81acf42c" path="/var/lib/kubelet/pods/33e5e73c-9714-463e-aeda-f83d81acf42c/volumes" Oct 31 02:10:18.499776 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:18.499740 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:10:22.709259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.709233 5756 generic.go:356] "Generic (PLEG): container finished" podID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerID="9ab2e62451c8245aa4a75c6b8d2bf452abc10e31acd9c539232d898169c4bf59" exitCode=0 Oct 31 02:10:22.709564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.709293 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" event={"ID":"81f1a671-4208-4d65-9eff-d9ba3be97be0","Type":"ContainerDied","Data":"9ab2e62451c8245aa4a75c6b8d2bf452abc10e31acd9c539232d898169c4bf59"} Oct 31 02:10:22.709564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.709330 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" event={"ID":"81f1a671-4208-4d65-9eff-d9ba3be97be0","Type":"ContainerDied","Data":"b45a56af7ee56006b6eb38b5d2205e2eddc5dca763fa6101209832ece17cbd8c"} Oct 31 02:10:22.709564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.709348 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b45a56af7ee56006b6eb38b5d2205e2eddc5dca763fa6101209832ece17cbd8c" Oct 31 02:10:22.722495 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.722470 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 02:10:22.799283 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.799249 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf\" (UniqueName: \"kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf\") pod \"81f1a671-4208-4d65-9eff-d9ba3be97be0\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " Oct 31 02:10:22.799477 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.799356 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume\") pod \"81f1a671-4208-4d65-9eff-d9ba3be97be0\" (UID: \"81f1a671-4208-4d65-9eff-d9ba3be97be0\") " Oct 31 02:10:22.799715 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.799693 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume" pod "81f1a671-4208-4d65-9eff-d9ba3be97be0" (UID: "81f1a671-4208-4d65-9eff-d9ba3be97be0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:22.799801 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.799780 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f1a671-4208-4d65-9eff-d9ba3be97be0-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:22.801476 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.801439 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf" pod "81f1a671-4208-4d65-9eff-d9ba3be97be0" (UID: "81f1a671-4208-4d65-9eff-d9ba3be97be0"). InnerVolumeSpecName "kube-api-access-sdtqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:22.901239 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:22.901201 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdtqf\" (UniqueName: \"kubernetes.io/projected/81f1a671-4208-4d65-9eff-d9ba3be97be0-kube-api-access-sdtqf\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:23.715148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:23.715111 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76" Oct 31 02:10:23.878979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:23.878945 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76"] Oct 31 02:10:23.883799 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:23.883764 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-9c9c3c-6dd578b7cf-nwj76"] Oct 31 02:10:24.311232 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.311200 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" path="/var/lib/kubelet/pods/81f1a671-4208-4d65-9eff-d9ba3be97be0/volumes" Oct 31 02:10:24.747983 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.747897 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx"] Oct 31 02:10:24.748433 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748417 5756 cpu_manager.go:414] "containerMap: removing container" podUID="184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" containerName="revoke-keys" Oct 31 02:10:24.748484 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748435 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" containerName="revoke-keys" Oct 31 02:10:24.748484 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748465 5756 cpu_manager.go:414] "containerMap: removing container" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerName="payload-processing" Oct 31 02:10:24.748484 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748471 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerName="payload-processing" Oct 31 02:10:24.748578 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748496 5756 cpu_manager.go:414] "containerMap: removing container" podUID="5d950f0d-a992-4c37-8ad3-08fa95bd91f9" containerName="revoke-keys" Oct 31 02:10:24.748578 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748502 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d950f0d-a992-4c37-8ad3-08fa95bd91f9" containerName="revoke-keys" Oct 31 02:10:24.748645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748586 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" containerName="revoke-keys" Oct 31 02:10:24.748645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748597 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f1a671-4208-4d65-9eff-d9ba3be97be0" containerName="payload-processing" Oct 31 02:10:24.748645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.748611 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d950f0d-a992-4c37-8ad3-08fa95bd91f9" containerName="revoke-keys" Oct 31 02:10:24.752012 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.751995 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.808129 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.808095 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx"] Oct 31 02:10:24.823117 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.823086 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bs7b\" (UniqueName: \"kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.823275 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.823201 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.823275 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.823239 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.854419 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.854389 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:10:24.854560 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.854472 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:10:24.924723 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.924690 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.924938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.924730 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.924938 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.924905 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bs7b\" (UniqueName: \"kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.925387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.925371 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.927340 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.927318 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:24.981609 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:24.981573 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bs7b\" (UniqueName: \"kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b\") pod \"maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:25.065303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:25.065271 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:25.456725 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:25.456705 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx"] Oct 31 02:10:25.459136 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:25.459104 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1f5081_3210_4db3_b491_f30c23b8dbcf.slice/crio-9f99482f0f1ed6428e14e62aa403c768bba302b68190e791444f2e4f844d3049 WatchSource:0}: Error finding container 9f99482f0f1ed6428e14e62aa403c768bba302b68190e791444f2e4f844d3049: Status 404 returned error can't find the container with id 9f99482f0f1ed6428e14e62aa403c768bba302b68190e791444f2e4f844d3049 Oct 31 02:10:25.729924 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:25.729812 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" event={"ID":"3a1f5081-3210-4db3-b491-f30c23b8dbcf","Type":"ContainerStarted","Data":"9f99482f0f1ed6428e14e62aa403c768bba302b68190e791444f2e4f844d3049"} Oct 31 02:10:26.748797 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:26.748708 5756 generic.go:356] "Generic (PLEG): container finished" podID="3a1f5081-3210-4db3-b491-f30c23b8dbcf" containerID="67517818d0287c8ac00609fca8adbd3ffa65062ae5b1f13488cd45e702918858" exitCode=0 Oct 31 02:10:26.749303 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:26.748792 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" event={"ID":"3a1f5081-3210-4db3-b491-f30c23b8dbcf","Type":"ContainerDied","Data":"67517818d0287c8ac00609fca8adbd3ffa65062ae5b1f13488cd45e702918858"} Oct 31 02:10:27.912400 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:27.912371 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:28.059179 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.059153 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca\") pod \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " Oct 31 02:10:28.059359 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.059222 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b\" (UniqueName: \"kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b\") pod \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " Oct 31 02:10:28.059448 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.059358 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca\") pod \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\" (UID: \"3a1f5081-3210-4db3-b491-f30c23b8dbcf\") " Oct 31 02:10:28.059542 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.059516 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca" pod "3a1f5081-3210-4db3-b491-f30c23b8dbcf" (UID: "3a1f5081-3210-4db3-b491-f30c23b8dbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:28.059717 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.059693 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:28.061604 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.061581 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca" pod "3a1f5081-3210-4db3-b491-f30c23b8dbcf" (UID: "3a1f5081-3210-4db3-b491-f30c23b8dbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:28.061689 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.061669 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b" pod "3a1f5081-3210-4db3-b491-f30c23b8dbcf" (UID: "3a1f5081-3210-4db3-b491-f30c23b8dbcf"). InnerVolumeSpecName "kube-api-access-7bs7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:28.160708 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.160672 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bs7b\" (UniqueName: \"kubernetes.io/projected/3a1f5081-3210-4db3-b491-f30c23b8dbcf-kube-api-access-7bs7b\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:28.160708 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.160706 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1f5081-3210-4db3-b491-f30c23b8dbcf-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:28.764678 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.764593 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" Oct 31 02:10:28.764678 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.764625 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx" event={"ID":"3a1f5081-3210-4db3-b491-f30c23b8dbcf","Type":"ContainerDied","Data":"9f99482f0f1ed6428e14e62aa403c768bba302b68190e791444f2e4f844d3049"} Oct 31 02:10:28.764678 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:28.764676 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f99482f0f1ed6428e14e62aa403c768bba302b68190e791444f2e4f844d3049" Oct 31 02:10:30.349826 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.349787 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" containerID="cri-o://2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987" gracePeriod=45 Oct 31 02:10:30.365281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.365233 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" containerID="cri-o://ecb4de90141f9d246aeadc803f430103513483b996d0256b2b28ee392bd743fd" gracePeriod=45 Oct 31 02:10:30.397490 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.397431 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" containerID="cri-o://320ff6ceda24321c65d26d97ee4bd193a3602669e90ba4cd6a41a19bcf052399" gracePeriod=45 Oct 31 02:10:30.675168 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.675127 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" containerID="cri-o://274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c" gracePeriod=45 Oct 31 02:10:30.770499 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.770475 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 02:10:30.779654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.779628 5756 generic.go:356] "Generic (PLEG): container finished" podID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerID="2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987" exitCode=0 Oct 31 02:10:30.779737 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.779687 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" Oct 31 02:10:30.779804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.779690 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" event={"ID":"e89f39e4-29e9-4374-b0ac-c88ae0836486","Type":"ContainerDied","Data":"2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987"} Oct 31 02:10:30.779904 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.779818 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj" event={"ID":"e89f39e4-29e9-4374-b0ac-c88ae0836486","Type":"ContainerDied","Data":"e066d1e9797a967844b41cbdc6e36a30521054330c2c09c72fdf0632209caa7c"} Oct 31 02:10:30.779904 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.779849 5756 scope.go:122] "RemoveContainer" containerID="2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987" Oct 31 02:10:30.781288 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.781269 5756 generic.go:356] "Generic (PLEG): container finished" podID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerID="ecb4de90141f9d246aeadc803f430103513483b996d0256b2b28ee392bd743fd" exitCode=0 Oct 31 02:10:30.781375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.781341 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" event={"ID":"398da614-85b5-47d4-8f6f-7f13bf3b145f","Type":"ContainerDied","Data":"ecb4de90141f9d246aeadc803f430103513483b996d0256b2b28ee392bd743fd"} Oct 31 02:10:30.782830 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.782808 5756 generic.go:356] "Generic (PLEG): container finished" podID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerID="320ff6ceda24321c65d26d97ee4bd193a3602669e90ba4cd6a41a19bcf052399" exitCode=0 Oct 31 02:10:30.782957 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.782896 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" event={"ID":"7266da26-c8f4-429e-8295-8f25b463a6f1","Type":"ContainerDied","Data":"320ff6ceda24321c65d26d97ee4bd193a3602669e90ba4cd6a41a19bcf052399"} Oct 31 02:10:30.789147 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.789088 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 02:10:30.790894 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.790856 5756 scope.go:122] "RemoveContainer" containerID="2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987" Oct 31 02:10:30.791189 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:10:30.791168 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987\": container with ID starting with 2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987 not found: ID does not exist" containerID="2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987" Oct 31 02:10:30.791260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.791199 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987"} err="failed to get container status \"2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987\": rpc error: code = NotFound desc = could not find container \"2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987\": container with ID starting with 2c9548d6ad781db463abee70b510f99c4ac94fa15ee72f18b32b027bae42e987 not found: ID does not exist" Oct 31 02:10:30.792914 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.792896 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 02:10:30.828255 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.828225 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": dial tcp 10.134.0.51:8000: connect: connection refused" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.890244 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home\") pod \"7266da26-c8f4-429e-8295-8f25b463a6f1\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.890300 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm\") pod \"7266da26-c8f4-429e-8295-8f25b463a6f1\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.890328 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache\") pod \"398da614-85b5-47d4-8f6f-7f13bf3b145f\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.890690 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache" pod "398da614-85b5-47d4-8f6f-7f13bf3b145f" (UID: "398da614-85b5-47d4-8f6f-7f13bf3b145f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891098 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home" pod "7266da26-c8f4-429e-8295-8f25b463a6f1" (UID: "7266da26-c8f4-429e-8295-8f25b463a6f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891199 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs\") pod \"7266da26-c8f4-429e-8295-8f25b463a6f1\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891234 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm\") pod \"e89f39e4-29e9-4374-b0ac-c88ae0836486\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891289 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home\") pod \"e89f39e4-29e9-4374-b0ac-c88ae0836486\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891339 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs\" (UniqueName: \"kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs\") pod \"398da614-85b5-47d4-8f6f-7f13bf3b145f\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891366 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir\") pod \"7266da26-c8f4-429e-8295-8f25b463a6f1\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891398 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home\") pod \"398da614-85b5-47d4-8f6f-7f13bf3b145f\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891443 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v\" (UniqueName: \"kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v\") pod \"e89f39e4-29e9-4374-b0ac-c88ae0836486\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891503 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir\") pod \"398da614-85b5-47d4-8f6f-7f13bf3b145f\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891531 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56\" (UniqueName: \"kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56\") pod \"398da614-85b5-47d4-8f6f-7f13bf3b145f\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891582 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache\") pod \"e89f39e4-29e9-4374-b0ac-c88ae0836486\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891622 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache\") pod \"7266da26-c8f4-429e-8295-8f25b463a6f1\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891662 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9\" (UniqueName: \"kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9\") pod \"7266da26-c8f4-429e-8295-8f25b463a6f1\" (UID: \"7266da26-c8f4-429e-8295-8f25b463a6f1\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891705 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm\") pod \"398da614-85b5-47d4-8f6f-7f13bf3b145f\" (UID: \"398da614-85b5-47d4-8f6f-7f13bf3b145f\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891729 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir\") pod \"e89f39e4-29e9-4374-b0ac-c88ae0836486\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.891800 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs\" (UniqueName: \"kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs\") pod \"e89f39e4-29e9-4374-b0ac-c88ae0836486\" (UID: \"e89f39e4-29e9-4374-b0ac-c88ae0836486\") " Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.892202 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache" pod "7266da26-c8f4-429e-8295-8f25b463a6f1" (UID: "7266da26-c8f4-429e-8295-8f25b463a6f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.892807 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home" pod "e89f39e4-29e9-4374-b0ac-c88ae0836486" (UID: "e89f39e4-29e9-4374-b0ac-c88ae0836486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.893126 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache" pod "e89f39e4-29e9-4374-b0ac-c88ae0836486" (UID: "e89f39e4-29e9-4374-b0ac-c88ae0836486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.894793 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir" pod "7266da26-c8f4-429e-8295-8f25b463a6f1" (UID: "7266da26-c8f4-429e-8295-8f25b463a6f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.896552 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm" pod "7266da26-c8f4-429e-8295-8f25b463a6f1" (UID: "7266da26-c8f4-429e-8295-8f25b463a6f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.896948 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir" pod "398da614-85b5-47d4-8f6f-7f13bf3b145f" (UID: "398da614-85b5-47d4-8f6f-7f13bf3b145f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.897135 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir" pod "e89f39e4-29e9-4374-b0ac-c88ae0836486" (UID: "e89f39e4-29e9-4374-b0ac-c88ae0836486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.898279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.897198 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home" pod "398da614-85b5-47d4-8f6f-7f13bf3b145f" (UID: "398da614-85b5-47d4-8f6f-7f13bf3b145f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.899828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.899384 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56" pod "398da614-85b5-47d4-8f6f-7f13bf3b145f" (UID: "398da614-85b5-47d4-8f6f-7f13bf3b145f"). InnerVolumeSpecName "kube-api-access-g8f56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:30.899828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.899384 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs" pod "398da614-85b5-47d4-8f6f-7f13bf3b145f" (UID: "398da614-85b5-47d4-8f6f-7f13bf3b145f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:30.899828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.899452 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs" pod "7266da26-c8f4-429e-8295-8f25b463a6f1" (UID: "7266da26-c8f4-429e-8295-8f25b463a6f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:30.899828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.899488 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9" pod "7266da26-c8f4-429e-8295-8f25b463a6f1" (UID: "7266da26-c8f4-429e-8295-8f25b463a6f1"). InnerVolumeSpecName "kube-api-access-zmzj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:30.899828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.899647 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm" pod "e89f39e4-29e9-4374-b0ac-c88ae0836486" (UID: "e89f39e4-29e9-4374-b0ac-c88ae0836486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.899828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.899774 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v" pod "e89f39e4-29e9-4374-b0ac-c88ae0836486" (UID: "e89f39e4-29e9-4374-b0ac-c88ae0836486"). InnerVolumeSpecName "kube-api-access-mvb4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:30.901069 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901041 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvb4v\" (UniqueName: \"kubernetes.io/projected/e89f39e4-29e9-4374-b0ac-c88ae0836486-kube-api-access-mvb4v\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901073 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901092 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8f56\" (UniqueName: \"kubernetes.io/projected/398da614-85b5-47d4-8f6f-7f13bf3b145f-kube-api-access-g8f56\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901108 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901122 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901137 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmzj9\" (UniqueName: \"kubernetes.io/projected/7266da26-c8f4-429e-8295-8f25b463a6f1-kube-api-access-zmzj9\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901151 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901163 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901175 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901188 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901201 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7266da26-c8f4-429e-8295-8f25b463a6f1-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901214 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901227 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e89f39e4-29e9-4374-b0ac-c88ae0836486-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901239 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/398da614-85b5-47d4-8f6f-7f13bf3b145f-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901253 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7266da26-c8f4-429e-8295-8f25b463a6f1-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.901270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.901266 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:30.902076 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.902053 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs" pod "e89f39e4-29e9-4374-b0ac-c88ae0836486" (UID: "e89f39e4-29e9-4374-b0ac-c88ae0836486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:30.902931 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.902905 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm" pod "398da614-85b5-47d4-8f6f-7f13bf3b145f" (UID: "398da614-85b5-47d4-8f6f-7f13bf3b145f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:30.926968 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:30.926901 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 02:10:31.002001 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.001946 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/398da614-85b5-47d4-8f6f-7f13bf3b145f-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.002001 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.001977 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e89f39e4-29e9-4374-b0ac-c88ae0836486-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.102458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102424 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs\") pod \"4cad83b9-9b46-4174-b37b-7663d23b67e1\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " Oct 31 02:10:31.102624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102485 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir\") pod \"4cad83b9-9b46-4174-b37b-7663d23b67e1\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " Oct 31 02:10:31.102624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102554 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98\" (UniqueName: \"kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98\") pod \"4cad83b9-9b46-4174-b37b-7663d23b67e1\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " Oct 31 02:10:31.102624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102597 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache\") pod \"4cad83b9-9b46-4174-b37b-7663d23b67e1\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " Oct 31 02:10:31.102786 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102651 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm\") pod \"4cad83b9-9b46-4174-b37b-7663d23b67e1\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " Oct 31 02:10:31.102786 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102683 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home\") pod \"4cad83b9-9b46-4174-b37b-7663d23b67e1\" (UID: \"4cad83b9-9b46-4174-b37b-7663d23b67e1\") " Oct 31 02:10:31.102926 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102782 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir" pod "4cad83b9-9b46-4174-b37b-7663d23b67e1" (UID: "4cad83b9-9b46-4174-b37b-7663d23b67e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:31.102985 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102948 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache" pod "4cad83b9-9b46-4174-b37b-7663d23b67e1" (UID: "4cad83b9-9b46-4174-b37b-7663d23b67e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:31.102985 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.102964 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home" pod "4cad83b9-9b46-4174-b37b-7663d23b67e1" (UID: "4cad83b9-9b46-4174-b37b-7663d23b67e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:31.103169 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.103151 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.103254 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.103171 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.103254 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.103180 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.104827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.104798 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm" pod "4cad83b9-9b46-4174-b37b-7663d23b67e1" (UID: "4cad83b9-9b46-4174-b37b-7663d23b67e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:31.104827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.104809 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs" pod "4cad83b9-9b46-4174-b37b-7663d23b67e1" (UID: "4cad83b9-9b46-4174-b37b-7663d23b67e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:31.105035 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.104906 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98" pod "4cad83b9-9b46-4174-b37b-7663d23b67e1" (UID: "4cad83b9-9b46-4174-b37b-7663d23b67e1"). InnerVolumeSpecName "kube-api-access-vzb98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:31.204765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.204666 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzb98\" (UniqueName: \"kubernetes.io/projected/4cad83b9-9b46-4174-b37b-7663d23b67e1-kube-api-access-vzb98\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.204765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.204697 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cad83b9-9b46-4174-b37b-7663d23b67e1-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.204765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.204707 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cad83b9-9b46-4174-b37b-7663d23b67e1-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:31.274940 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.274907 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj"] Oct 31 02:10:31.284397 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.280127 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-2-w2-e621db-kserve-54bcbbdb8d-4ghdj"] Oct 31 02:10:31.671979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.671922 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 02:10:31.791176 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.791140 5756 generic.go:356] "Generic (PLEG): container finished" podID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerID="274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c" exitCode=0 Oct 31 02:10:31.791358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.791196 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" Oct 31 02:10:31.791358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.791217 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" event={"ID":"4cad83b9-9b46-4174-b37b-7663d23b67e1","Type":"ContainerDied","Data":"274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c"} Oct 31 02:10:31.791358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.791252 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n" event={"ID":"4cad83b9-9b46-4174-b37b-7663d23b67e1","Type":"ContainerDied","Data":"7a746cb542518da590619c08d463f03b4b47d9074ef4b14d26797b841e59107d"} Oct 31 02:10:31.791358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.791268 5756 scope.go:122] "RemoveContainer" containerID="274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c" Oct 31 02:10:31.792765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.792742 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" Oct 31 02:10:31.792948 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.792813 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8" event={"ID":"398da614-85b5-47d4-8f6f-7f13bf3b145f","Type":"ContainerDied","Data":"e65938e8e1d980748fa28799bf55fa25ada053fd8310d0c58b0037bde29f5fce"} Oct 31 02:10:31.795221 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.795196 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" Oct 31 02:10:31.795343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.795222 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7" event={"ID":"7266da26-c8f4-429e-8295-8f25b463a6f1","Type":"ContainerDied","Data":"e57e546b2843e177af239050bc5d350c10b10d7c60a724adbdfb613cea09b572"} Oct 31 02:10:31.808956 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.808937 5756 scope.go:122] "RemoveContainer" containerID="274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c" Oct 31 02:10:31.809391 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:10:31.809359 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c\": container with ID starting with 274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c not found: ID does not exist" containerID="274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c" Oct 31 02:10:31.809441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.809403 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c"} err="failed to get container status \"274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c\": rpc error: code = NotFound desc = could not find container \"274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c\": container with ID starting with 274ac344c95a2a0a3d31a9e0e61ebb11f23e31366156429f52052e23a4bed49c not found: ID does not exist" Oct 31 02:10:31.809441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.809425 5756 scope.go:122] "RemoveContainer" containerID="ecb4de90141f9d246aeadc803f430103513483b996d0256b2b28ee392bd743fd" Oct 31 02:10:31.818850 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.818833 5756 scope.go:122] "RemoveContainer" containerID="320ff6ceda24321c65d26d97ee4bd193a3602669e90ba4cd6a41a19bcf052399" Oct 31 02:10:31.953531 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.953452 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n"] Oct 31 02:10:31.963767 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:31.963737 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/simulator-w2-e621db-kserve-55667677cc-qrd9n"] Oct 31 02:10:32.067822 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:10:32.067789 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:10:32.071522 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.071497 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8"] Oct 31 02:10:32.075210 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.075186 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/distinct-w2-e621db-kserve-6fddbd456c-w7gf8"] Oct 31 02:10:32.186700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.186664 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7"] Oct 31 02:10:32.193433 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.193395 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e621db/premium-w2-e621db-kserve-5b9dcd847b-9htn7"] Oct 31 02:10:32.310225 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.310194 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" path="/var/lib/kubelet/pods/398da614-85b5-47d4-8f6f-7f13bf3b145f/volumes" Oct 31 02:10:32.310642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.310627 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" path="/var/lib/kubelet/pods/4cad83b9-9b46-4174-b37b-7663d23b67e1/volumes" Oct 31 02:10:32.311011 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.310997 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" path="/var/lib/kubelet/pods/7266da26-c8f4-429e-8295-8f25b463a6f1/volumes" Oct 31 02:10:32.311656 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:32.311642 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" path="/var/lib/kubelet/pods/e89f39e4-29e9-4374-b0ac-c88ae0836486/volumes" Oct 31 02:10:36.291216 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291183 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk"] Oct 31 02:10:36.291665 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291651 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3a1f5081-3210-4db3-b491-f30c23b8dbcf" containerName="revoke-keys" Oct 31 02:10:36.291710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291667 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1f5081-3210-4db3-b491-f30c23b8dbcf" containerName="revoke-keys" Oct 31 02:10:36.291710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291680 5756 cpu_manager.go:414] "containerMap: removing container" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" Oct 31 02:10:36.291710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291685 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" Oct 31 02:10:36.291710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291694 5756 cpu_manager.go:414] "containerMap: removing container" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" Oct 31 02:10:36.291710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291699 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" Oct 31 02:10:36.291710 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291709 5756 cpu_manager.go:414] "containerMap: removing container" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291715 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291737 5756 cpu_manager.go:414] "containerMap: removing container" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291742 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291822 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1f5081-3210-4db3-b491-f30c23b8dbcf" containerName="revoke-keys" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291832 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="7266da26-c8f4-429e-8295-8f25b463a6f1" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291838 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cad83b9-9b46-4174-b37b-7663d23b67e1" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291844 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="398da614-85b5-47d4-8f6f-7f13bf3b145f" containerName="main" Oct 31 02:10:36.291923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.291855 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89f39e4-29e9-4374-b0ac-c88ae0836486" containerName="main" Oct 31 02:10:36.296501 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.296484 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.352315 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.352285 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm"] Oct 31 02:10:36.356046 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.356028 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.399119 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.399091 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:10:36.399276 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.399123 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:10:36.401153 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.401134 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk"] Oct 31 02:10:36.412311 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.412249 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm"] Oct 31 02:10:36.461174 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.461144 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxzz\" (UniqueName: \"kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.461361 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.461183 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.461361 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.461204 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v25f4\" (UniqueName: \"kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.461361 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.461343 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.461493 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.461374 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.461493 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.461417 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.562387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.562297 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.562387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.562356 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.562387 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.562396 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.562693 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.562458 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxzz\" (UniqueName: \"kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.562693 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.562480 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.562693 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.562498 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v25f4\" (UniqueName: \"kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.563137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.563110 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.563421 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.563174 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.564834 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.564812 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.564979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.564963 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.619439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.619410 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v25f4\" (UniqueName: \"kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4\") pod \"maas-api-revoke-sub-6d2bd76d-2mjvm\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.619580 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.619491 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxzz\" (UniqueName: \"kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz\") pod \"maas-api-revoke-sub-c1794932-4rvgk\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:36.670939 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.670907 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:36.909911 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:36.909816 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:37.044716 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.044687 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm"] Oct 31 02:10:37.046110 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:37.046081 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3479d69c_2d01_4b7a_adfc_1c3ce2459a47.slice/crio-526170179b61fcefa7227f69f152ee46f3a04bc21a8db4bcfe4c129234db37b5 WatchSource:0}: Error finding container 526170179b61fcefa7227f69f152ee46f3a04bc21a8db4bcfe4c129234db37b5: Status 404 returned error can't find the container with id 526170179b61fcefa7227f69f152ee46f3a04bc21a8db4bcfe4c129234db37b5 Oct 31 02:10:37.282779 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.282749 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk"] Oct 31 02:10:37.284856 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:37.284826 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc396070_7cf4_4868_8a62_6ce18722318c.slice/crio-1208202e05546ce2cd90da195d2f773a0b0cd5d1c1585c4e9bc143c596a58508 WatchSource:0}: Error finding container 1208202e05546ce2cd90da195d2f773a0b0cd5d1c1585c4e9bc143c596a58508: Status 404 returned error can't find the container with id 1208202e05546ce2cd90da195d2f773a0b0cd5d1c1585c4e9bc143c596a58508 Oct 31 02:10:37.841213 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.841128 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" event={"ID":"3479d69c-2d01-4b7a-adfc-1c3ce2459a47","Type":"ContainerStarted","Data":"2a1e2773f2dc7c794ed5757007655bbb66db4a02cec40c9e6fe9d658d0c45ecc"} Oct 31 02:10:37.841213 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.841190 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" event={"ID":"3479d69c-2d01-4b7a-adfc-1c3ce2459a47","Type":"ContainerStarted","Data":"526170179b61fcefa7227f69f152ee46f3a04bc21a8db4bcfe4c129234db37b5"} Oct 31 02:10:37.842825 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.842798 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" event={"ID":"fc396070-7cf4-4868-8a62-6ce18722318c","Type":"ContainerStarted","Data":"38db32335a2b0e6cdaf1ca7da00c75e42c9a04468aaa5527dbb09ccadccf2ecd"} Oct 31 02:10:37.842990 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.842849 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" event={"ID":"fc396070-7cf4-4868-8a62-6ce18722318c","Type":"ContainerStarted","Data":"1208202e05546ce2cd90da195d2f773a0b0cd5d1c1585c4e9bc143c596a58508"} Oct 31 02:10:37.954182 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:37.954121 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" podStartSLOduration=1.954100034 podStartE2EDuration="1.954100034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:37.952683256 +0000 UTC m=+4409.759650801" watchObservedRunningTime="2026-10-31 02:10:37.954100034 +0000 UTC m=+4409.761067579" Oct 31 02:10:38.067855 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:38.067795 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" podStartSLOduration=2.067776349 podStartE2EDuration="2.067776349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:38.066129985 +0000 UTC m=+4409.873097528" watchObservedRunningTime="2026-10-31 02:10:38.067776349 +0000 UTC m=+4409.874743895" Oct 31 02:10:38.852042 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:38.852005 5756 generic.go:356] "Generic (PLEG): container finished" podID="3479d69c-2d01-4b7a-adfc-1c3ce2459a47" containerID="2a1e2773f2dc7c794ed5757007655bbb66db4a02cec40c9e6fe9d658d0c45ecc" exitCode=0 Oct 31 02:10:38.852527 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:38.852100 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" event={"ID":"3479d69c-2d01-4b7a-adfc-1c3ce2459a47","Type":"ContainerDied","Data":"2a1e2773f2dc7c794ed5757007655bbb66db4a02cec40c9e6fe9d658d0c45ecc"} Oct 31 02:10:38.853849 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:38.853825 5756 generic.go:356] "Generic (PLEG): container finished" podID="fc396070-7cf4-4868-8a62-6ce18722318c" containerID="38db32335a2b0e6cdaf1ca7da00c75e42c9a04468aaa5527dbb09ccadccf2ecd" exitCode=0 Oct 31 02:10:38.853994 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:38.853915 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" event={"ID":"fc396070-7cf4-4868-8a62-6ce18722318c","Type":"ContainerDied","Data":"38db32335a2b0e6cdaf1ca7da00c75e42c9a04468aaa5527dbb09ccadccf2ecd"} Oct 31 02:10:39.098281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:39.098245 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:10:40.015109 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.015087 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:40.017352 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.017336 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:40.100712 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.100677 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4\" (UniqueName: \"kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4\") pod \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " Oct 31 02:10:40.100918 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.100761 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca\") pod \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " Oct 31 02:10:40.100918 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.100786 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca\") pod \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\" (UID: \"3479d69c-2d01-4b7a-adfc-1c3ce2459a47\") " Oct 31 02:10:40.100918 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.100828 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca\") pod \"fc396070-7cf4-4868-8a62-6ce18722318c\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " Oct 31 02:10:40.100918 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.100915 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca\") pod \"fc396070-7cf4-4868-8a62-6ce18722318c\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " Oct 31 02:10:40.101161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.100977 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz\" (UniqueName: \"kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz\") pod \"fc396070-7cf4-4868-8a62-6ce18722318c\" (UID: \"fc396070-7cf4-4868-8a62-6ce18722318c\") " Oct 31 02:10:40.101226 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.101187 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca" pod "3479d69c-2d01-4b7a-adfc-1c3ce2459a47" (UID: "3479d69c-2d01-4b7a-adfc-1c3ce2459a47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:40.101312 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.101289 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:40.101364 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.101289 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca" pod "fc396070-7cf4-4868-8a62-6ce18722318c" (UID: "fc396070-7cf4-4868-8a62-6ce18722318c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:40.103143 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.103120 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca" pod "fc396070-7cf4-4868-8a62-6ce18722318c" (UID: "fc396070-7cf4-4868-8a62-6ce18722318c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:40.103277 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.103258 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz" pod "fc396070-7cf4-4868-8a62-6ce18722318c" (UID: "fc396070-7cf4-4868-8a62-6ce18722318c"). InnerVolumeSpecName "kube-api-access-tlxzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:40.103513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.103489 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca" pod "3479d69c-2d01-4b7a-adfc-1c3ce2459a47" (UID: "3479d69c-2d01-4b7a-adfc-1c3ce2459a47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:40.103513 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.103504 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4" pod "3479d69c-2d01-4b7a-adfc-1c3ce2459a47" (UID: "3479d69c-2d01-4b7a-adfc-1c3ce2459a47"). InnerVolumeSpecName "kube-api-access-v25f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:40.202199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.202113 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:40.202199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.202141 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:40.202199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.202152 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc396070-7cf4-4868-8a62-6ce18722318c-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:40.202199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.202161 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxzz\" (UniqueName: \"kubernetes.io/projected/fc396070-7cf4-4868-8a62-6ce18722318c-kube-api-access-tlxzz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:40.202199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.202170 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v25f4\" (UniqueName: \"kubernetes.io/projected/3479d69c-2d01-4b7a-adfc-1c3ce2459a47-kube-api-access-v25f4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:40.875041 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.875007 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" Oct 31 02:10:40.875256 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.875035 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm" event={"ID":"3479d69c-2d01-4b7a-adfc-1c3ce2459a47","Type":"ContainerDied","Data":"526170179b61fcefa7227f69f152ee46f3a04bc21a8db4bcfe4c129234db37b5"} Oct 31 02:10:40.875256 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.875115 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="526170179b61fcefa7227f69f152ee46f3a04bc21a8db4bcfe4c129234db37b5" Oct 31 02:10:40.876596 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.876578 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" Oct 31 02:10:40.876728 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.876577 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk" event={"ID":"fc396070-7cf4-4868-8a62-6ce18722318c","Type":"ContainerDied","Data":"1208202e05546ce2cd90da195d2f773a0b0cd5d1c1585c4e9bc143c596a58508"} Oct 31 02:10:40.876728 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:40.876702 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1208202e05546ce2cd90da195d2f773a0b0cd5d1c1585c4e9bc143c596a58508" Oct 31 02:10:51.522670 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.522596 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz"] Oct 31 02:10:51.523105 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.523091 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3479d69c-2d01-4b7a-adfc-1c3ce2459a47" containerName="revoke-keys" Oct 31 02:10:51.523154 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.523106 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3479d69c-2d01-4b7a-adfc-1c3ce2459a47" containerName="revoke-keys" Oct 31 02:10:51.523154 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.523125 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fc396070-7cf4-4868-8a62-6ce18722318c" containerName="revoke-keys" Oct 31 02:10:51.523154 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.523130 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc396070-7cf4-4868-8a62-6ce18722318c" containerName="revoke-keys" Oct 31 02:10:51.523250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.523197 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3479d69c-2d01-4b7a-adfc-1c3ce2459a47" containerName="revoke-keys" Oct 31 02:10:51.523250 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.523209 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc396070-7cf4-4868-8a62-6ce18722318c" containerName="revoke-keys" Oct 31 02:10:51.526496 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.526474 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.567722 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.567695 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl"] Oct 31 02:10:51.571348 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.571330 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.580001 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.579330 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:10:51.615014 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.614992 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:10:51.617187 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.617160 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz"] Oct 31 02:10:51.632376 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.632353 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.632517 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.632400 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7nv8\" (UniqueName: \"kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.632586 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.632542 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.632661 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.632642 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.632724 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.632678 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.632784 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.632755 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5mtp\" (UniqueName: \"kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.644261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.644231 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl"] Oct 31 02:10:51.733652 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.733617 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5mtp\" (UniqueName: \"kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.733893 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.733697 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.733893 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.733755 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7nv8\" (UniqueName: \"kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.734454 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.733836 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.734547 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.734488 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.734600 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.734544 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.734600 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.734573 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.735343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.735317 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.736032 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.736011 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz"] Oct 31 02:10:51.736968 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.736797 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.737068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.737055 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.740073 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.740057 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.793276 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.793240 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5mtp\" (UniqueName: \"kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp\") pod \"maas-api-revoke-sub-ced32f81-wjtfl\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.793439 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.793244 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7nv8\" (UniqueName: \"kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8\") pod \"maas-api-revoke-sub-3ffe7285-vtvwz\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.793952 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.793931 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6"] Oct 31 02:10:51.798517 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.798493 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz"] Oct 31 02:10:51.798673 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.798597 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:51.836183 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.836152 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2ldm\" (UniqueName: \"kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.836358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.836206 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.836358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.836340 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.839913 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.839893 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:51.864265 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.863487 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6"] Oct 31 02:10:51.884399 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.884372 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:51.948034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.937788 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2ldm\" (UniqueName: \"kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.948034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.937875 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npl9j\" (UniqueName: \"kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:51.948034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.937927 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.948034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.938003 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:51.948034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.938027 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.948034 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.938098 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:51.948800 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.948775 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.956564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.956531 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:51.999517 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:51.999482 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2ldm\" (UniqueName: \"kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm\") pod \"maas-api-revoke-sub-6bcafe23-drvzz\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:52.039478 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.039447 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.039631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.039537 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npl9j\" (UniqueName: \"kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.039631 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.039595 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.040098 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.040077 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.042011 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.041990 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.066061 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.066003 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:52.097556 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.097530 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npl9j\" (UniqueName: \"kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j\") pod \"maas-api-revoke-sub-1ba2ca5f-l7tn6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.125216 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.125178 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:52.225949 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.225923 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz"] Oct 31 02:10:52.227927 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:52.227897 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb047bf2e_8097_4295_995f_338a8fcacac6.slice/crio-5242bc0408cb9957ebbfb24fe117cfd45587a12d245bb2e4f89a10b35c02b4ba WatchSource:0}: Error finding container 5242bc0408cb9957ebbfb24fe117cfd45587a12d245bb2e4f89a10b35c02b4ba: Status 404 returned error can't find the container with id 5242bc0408cb9957ebbfb24fe117cfd45587a12d245bb2e4f89a10b35c02b4ba Oct 31 02:10:52.275096 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.275074 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl"] Oct 31 02:10:52.276626 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:52.276603 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b13744_e69d_45d4_a8f4_c7ec207b29f8.slice/crio-036519b88218a5c9475c48f8ce11dd5fec190619f56d2243813d218aabdbb286 WatchSource:0}: Error finding container 036519b88218a5c9475c48f8ce11dd5fec190619f56d2243813d218aabdbb286: Status 404 returned error can't find the container with id 036519b88218a5c9475c48f8ce11dd5fec190619f56d2243813d218aabdbb286 Oct 31 02:10:52.442437 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.442412 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz"] Oct 31 02:10:52.446153 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:52.446126 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f2ed87_350e_4306_a7cc_81c17a3dd2d0.slice/crio-5e6ec0063cd4add69c3615532db43228a95963f885e9a6cbbaa6822144d1a930 WatchSource:0}: Error finding container 5e6ec0063cd4add69c3615532db43228a95963f885e9a6cbbaa6822144d1a930: Status 404 returned error can't find the container with id 5e6ec0063cd4add69c3615532db43228a95963f885e9a6cbbaa6822144d1a930 Oct 31 02:10:52.507705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.507685 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6"] Oct 31 02:10:52.509664 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:10:52.509640 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3035265c_b1fc_4ff6_b025_f02efc0f39a6.slice/crio-f1f7bdfdc2fd56b7ef2c2bdd79e5421c8082af3ae3830f63162b86e63ef9d356 WatchSource:0}: Error finding container f1f7bdfdc2fd56b7ef2c2bdd79e5421c8082af3ae3830f63162b86e63ef9d356: Status 404 returned error can't find the container with id f1f7bdfdc2fd56b7ef2c2bdd79e5421c8082af3ae3830f63162b86e63ef9d356 Oct 31 02:10:52.974541 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.974503 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" event={"ID":"3035265c-b1fc-4ff6-b025-f02efc0f39a6","Type":"ContainerStarted","Data":"4b812efff954d4a9708ac1ee5b87483d02588302ddececc92032444a71ae217d"} Oct 31 02:10:52.974965 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.974582 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" event={"ID":"3035265c-b1fc-4ff6-b025-f02efc0f39a6","Type":"ContainerStarted","Data":"f1f7bdfdc2fd56b7ef2c2bdd79e5421c8082af3ae3830f63162b86e63ef9d356"} Oct 31 02:10:52.977126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.977091 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" event={"ID":"a9b13744-e69d-45d4-a8f4-c7ec207b29f8","Type":"ContainerStarted","Data":"9ad3676f94110e06f70f63b80da761f0e5d3da8fb0b3a6406aa03e6aeb7d06a3"} Oct 31 02:10:52.977277 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.977148 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" event={"ID":"a9b13744-e69d-45d4-a8f4-c7ec207b29f8","Type":"ContainerStarted","Data":"036519b88218a5c9475c48f8ce11dd5fec190619f56d2243813d218aabdbb286"} Oct 31 02:10:52.979178 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.979150 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" event={"ID":"b047bf2e-8097-4295-995f-338a8fcacac6","Type":"ContainerStarted","Data":"446dd0dc0fa2cc030202f23e873a53c9be317ceab5c1b072a6ad29fa5ea10971"} Oct 31 02:10:52.979300 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.979201 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" event={"ID":"b047bf2e-8097-4295-995f-338a8fcacac6","Type":"ContainerStarted","Data":"5242bc0408cb9957ebbfb24fe117cfd45587a12d245bb2e4f89a10b35c02b4ba"} Oct 31 02:10:52.980972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.980938 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" event={"ID":"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0","Type":"ContainerStarted","Data":"71e6b56bc4aafdfda1c1fd5cb95867e2dba1f528e3740a4f5f4a8b0e87dd0bfb"} Oct 31 02:10:52.981106 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:52.980984 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" event={"ID":"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0","Type":"ContainerStarted","Data":"5e6ec0063cd4add69c3615532db43228a95963f885e9a6cbbaa6822144d1a930"} Oct 31 02:10:53.088944 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.088853 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" podStartSLOduration=2.088836057 podStartE2EDuration="2.088836057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:53.087448826 +0000 UTC m=+4424.894416372" watchObservedRunningTime="2026-10-31 02:10:53.088836057 +0000 UTC m=+4424.895803599" Oct 31 02:10:53.202902 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.202818 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" podStartSLOduration=2.202797452 podStartE2EDuration="2.202797452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:53.202220881 +0000 UTC m=+4425.009188425" watchObservedRunningTime="2026-10-31 02:10:53.202797452 +0000 UTC m=+4425.009764996" Oct 31 02:10:53.317519 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.317458 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" podStartSLOduration=2.317442453 podStartE2EDuration="2.317442453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:53.316998989 +0000 UTC m=+4425.123966534" watchObservedRunningTime="2026-10-31 02:10:53.317442453 +0000 UTC m=+4425.124410013" Oct 31 02:10:53.430078 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.430021 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" podStartSLOduration=2.430002461 podStartE2EDuration="2.430002461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:53.429338449 +0000 UTC m=+4425.236305995" watchObservedRunningTime="2026-10-31 02:10:53.430002461 +0000 UTC m=+4425.236970005" Oct 31 02:10:53.989470 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.989435 5756 generic.go:356] "Generic (PLEG): container finished" podID="b047bf2e-8097-4295-995f-338a8fcacac6" containerID="446dd0dc0fa2cc030202f23e873a53c9be317ceab5c1b072a6ad29fa5ea10971" exitCode=0 Oct 31 02:10:53.989980 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.989511 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" event={"ID":"b047bf2e-8097-4295-995f-338a8fcacac6","Type":"ContainerDied","Data":"446dd0dc0fa2cc030202f23e873a53c9be317ceab5c1b072a6ad29fa5ea10971"} Oct 31 02:10:53.991075 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.991055 5756 generic.go:356] "Generic (PLEG): container finished" podID="c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" containerID="71e6b56bc4aafdfda1c1fd5cb95867e2dba1f528e3740a4f5f4a8b0e87dd0bfb" exitCode=0 Oct 31 02:10:53.991167 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.991144 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" event={"ID":"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0","Type":"ContainerDied","Data":"71e6b56bc4aafdfda1c1fd5cb95867e2dba1f528e3740a4f5f4a8b0e87dd0bfb"} Oct 31 02:10:53.992720 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.992694 5756 generic.go:356] "Generic (PLEG): container finished" podID="3035265c-b1fc-4ff6-b025-f02efc0f39a6" containerID="4b812efff954d4a9708ac1ee5b87483d02588302ddececc92032444a71ae217d" exitCode=0 Oct 31 02:10:53.992827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.992782 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" event={"ID":"3035265c-b1fc-4ff6-b025-f02efc0f39a6","Type":"ContainerDied","Data":"4b812efff954d4a9708ac1ee5b87483d02588302ddececc92032444a71ae217d"} Oct 31 02:10:53.994219 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.994201 5756 generic.go:356] "Generic (PLEG): container finished" podID="a9b13744-e69d-45d4-a8f4-c7ec207b29f8" containerID="9ad3676f94110e06f70f63b80da761f0e5d3da8fb0b3a6406aa03e6aeb7d06a3" exitCode=0 Oct 31 02:10:53.994313 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:53.994228 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" event={"ID":"a9b13744-e69d-45d4-a8f4-c7ec207b29f8","Type":"ContainerDied","Data":"9ad3676f94110e06f70f63b80da761f0e5d3da8fb0b3a6406aa03e6aeb7d06a3"} Oct 31 02:10:55.198056 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.197290 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:55.203672 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.203645 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:55.241340 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.241312 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:55.244585 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.244559 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.277011 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca\") pod \"b047bf2e-8097-4295-995f-338a8fcacac6\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.277077 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j\" (UniqueName: \"kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j\") pod \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.277147 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca\") pod \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.277269 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca\") pod \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\" (UID: \"3035265c-b1fc-4ff6-b025-f02efc0f39a6\") " Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.277372 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca\") pod \"b047bf2e-8097-4295-995f-338a8fcacac6\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.277428 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8\" (UniqueName: \"kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8\") pod \"b047bf2e-8097-4295-995f-338a8fcacac6\" (UID: \"b047bf2e-8097-4295-995f-338a8fcacac6\") " Oct 31 02:10:55.278654 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.278431 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca" pod "3035265c-b1fc-4ff6-b025-f02efc0f39a6" (UID: "3035265c-b1fc-4ff6-b025-f02efc0f39a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:55.279674 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.279636 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca" pod "b047bf2e-8097-4295-995f-338a8fcacac6" (UID: "b047bf2e-8097-4295-995f-338a8fcacac6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:55.283075 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.283043 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca" pod "3035265c-b1fc-4ff6-b025-f02efc0f39a6" (UID: "3035265c-b1fc-4ff6-b025-f02efc0f39a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:55.283218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.283079 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca" pod "b047bf2e-8097-4295-995f-338a8fcacac6" (UID: "b047bf2e-8097-4295-995f-338a8fcacac6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:55.283218 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.283187 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8" pod "b047bf2e-8097-4295-995f-338a8fcacac6" (UID: "b047bf2e-8097-4295-995f-338a8fcacac6"). InnerVolumeSpecName "kube-api-access-j7nv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:55.283486 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.283273 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j" pod "3035265c-b1fc-4ff6-b025-f02efc0f39a6" (UID: "3035265c-b1fc-4ff6-b025-f02efc0f39a6"). InnerVolumeSpecName "kube-api-access-npl9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.379379 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca\") pod \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.379524 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca\") pod \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.379567 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca\") pod \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.379603 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca\") pod \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.379645 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp\" (UniqueName: \"kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp\") pod \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\" (UID: \"a9b13744-e69d-45d4-a8f4-c7ec207b29f8\") " Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.379686 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm\" (UniqueName: \"kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm\") pod \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\" (UID: \"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0\") " Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.380296 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.380321 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npl9j\" (UniqueName: \"kubernetes.io/projected/3035265c-b1fc-4ff6-b025-f02efc0f39a6-kube-api-access-npl9j\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.380337 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.380352 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3035265c-b1fc-4ff6-b025-f02efc0f39a6-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.380366 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b047bf2e-8097-4295-995f-338a8fcacac6-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.380880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.380382 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7nv8\" (UniqueName: \"kubernetes.io/projected/b047bf2e-8097-4295-995f-338a8fcacac6-kube-api-access-j7nv8\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.383112 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.382735 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca" pod "c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" (UID: "c2f2ed87-350e-4306-a7cc-81c17a3dd2d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:55.383112 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.383011 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca" pod "a9b13744-e69d-45d4-a8f4-c7ec207b29f8" (UID: "a9b13744-e69d-45d4-a8f4-c7ec207b29f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:55.385684 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.385622 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca" pod "a9b13744-e69d-45d4-a8f4-c7ec207b29f8" (UID: "a9b13744-e69d-45d4-a8f4-c7ec207b29f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:55.385684 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.385639 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca" pod "c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" (UID: "c2f2ed87-350e-4306-a7cc-81c17a3dd2d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:10:55.386326 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.386300 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp" pod "a9b13744-e69d-45d4-a8f4-c7ec207b29f8" (UID: "a9b13744-e69d-45d4-a8f4-c7ec207b29f8"). InnerVolumeSpecName "kube-api-access-l5mtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:55.388624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.388596 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm" pod "c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" (UID: "c2f2ed87-350e-4306-a7cc-81c17a3dd2d0"). InnerVolumeSpecName "kube-api-access-s2ldm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:55.481695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.481646 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.481695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.481675 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.481695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.481687 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.481695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.481696 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.481695 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.481705 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5mtp\" (UniqueName: \"kubernetes.io/projected/a9b13744-e69d-45d4-a8f4-c7ec207b29f8-kube-api-access-l5mtp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:55.482114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:55.481716 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2ldm\" (UniqueName: \"kubernetes.io/projected/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0-kube-api-access-s2ldm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:10:56.010802 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.010770 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" Oct 31 02:10:56.011024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.010793 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6" event={"ID":"3035265c-b1fc-4ff6-b025-f02efc0f39a6","Type":"ContainerDied","Data":"f1f7bdfdc2fd56b7ef2c2bdd79e5421c8082af3ae3830f63162b86e63ef9d356"} Oct 31 02:10:56.011024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.010849 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1f7bdfdc2fd56b7ef2c2bdd79e5421c8082af3ae3830f63162b86e63ef9d356" Oct 31 02:10:56.012375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.012357 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" Oct 31 02:10:56.012512 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.012380 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl" event={"ID":"a9b13744-e69d-45d4-a8f4-c7ec207b29f8","Type":"ContainerDied","Data":"036519b88218a5c9475c48f8ce11dd5fec190619f56d2243813d218aabdbb286"} Oct 31 02:10:56.012512 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.012421 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036519b88218a5c9475c48f8ce11dd5fec190619f56d2243813d218aabdbb286" Oct 31 02:10:56.014029 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.013929 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" Oct 31 02:10:56.014029 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.013955 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz" event={"ID":"b047bf2e-8097-4295-995f-338a8fcacac6","Type":"ContainerDied","Data":"5242bc0408cb9957ebbfb24fe117cfd45587a12d245bb2e4f89a10b35c02b4ba"} Oct 31 02:10:56.014029 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.014000 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5242bc0408cb9957ebbfb24fe117cfd45587a12d245bb2e4f89a10b35c02b4ba" Oct 31 02:10:56.019899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.015600 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" event={"ID":"c2f2ed87-350e-4306-a7cc-81c17a3dd2d0","Type":"ContainerDied","Data":"5e6ec0063cd4add69c3615532db43228a95963f885e9a6cbbaa6822144d1a930"} Oct 31 02:10:56.019899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.015637 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz" Oct 31 02:10:56.019899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:10:56.015643 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e6ec0063cd4add69c3615532db43228a95963f885e9a6cbbaa6822144d1a930" Oct 31 02:11:02.069336 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:11:02.069302 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:11:11.523724 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:11.523691 5756 scope.go:122] "RemoveContainer" containerID="ecc848a1b4f0a563e9cafc0b5f22197e0b9f04ad7d807f244beb632bf593cc0f" Oct 31 02:11:11.534053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:11.533984 5756 scope.go:122] "RemoveContainer" containerID="b56311293cdb5aa3aff32f731ca68761f56ea14b6c7b3b36ee346add9b27b487" Oct 31 02:11:11.546346 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:11.546328 5756 scope.go:122] "RemoveContainer" containerID="9ab2e62451c8245aa4a75c6b8d2bf452abc10e31acd9c539232d898169c4bf59" Oct 31 02:11:11.555316 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:11.555294 5756 scope.go:122] "RemoveContainer" containerID="ded4968e1bf496379174cc829d9d3d1cf809dceea6fa16a69fd5ec216132f188" Oct 31 02:11:13.206249 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:13.206212 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7"] Oct 31 02:11:13.209813 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:13.209786 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-97b317-tl6k7"] Oct 31 02:11:13.256003 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:13.255971 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:11:14.310880 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:14.310840 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d950f0d-a992-4c37-8ad3-08fa95bd91f9" path="/var/lib/kubelet/pods/5d950f0d-a992-4c37-8ad3-08fa95bd91f9/volumes" Oct 31 02:11:28.224740 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.224698 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q"] Oct 31 02:11:28.225292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225241 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b047bf2e-8097-4295-995f-338a8fcacac6" containerName="revoke-keys" Oct 31 02:11:28.225292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225255 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b047bf2e-8097-4295-995f-338a8fcacac6" containerName="revoke-keys" Oct 31 02:11:28.225292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225276 5756 cpu_manager.go:414] "containerMap: removing container" podUID="c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" containerName="revoke-keys" Oct 31 02:11:28.225292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225281 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" containerName="revoke-keys" Oct 31 02:11:28.225292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225288 5756 cpu_manager.go:414] "containerMap: removing container" podUID="a9b13744-e69d-45d4-a8f4-c7ec207b29f8" containerName="revoke-keys" Oct 31 02:11:28.225292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225294 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b13744-e69d-45d4-a8f4-c7ec207b29f8" containerName="revoke-keys" Oct 31 02:11:28.225500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225317 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3035265c-b1fc-4ff6-b025-f02efc0f39a6" containerName="revoke-keys" Oct 31 02:11:28.225500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225321 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3035265c-b1fc-4ff6-b025-f02efc0f39a6" containerName="revoke-keys" Oct 31 02:11:28.225500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225405 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3035265c-b1fc-4ff6-b025-f02efc0f39a6" containerName="revoke-keys" Oct 31 02:11:28.225500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225418 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b047bf2e-8097-4295-995f-338a8fcacac6" containerName="revoke-keys" Oct 31 02:11:28.225500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225426 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" containerName="revoke-keys" Oct 31 02:11:28.225500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.225431 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b13744-e69d-45d4-a8f4-c7ec207b29f8" containerName="revoke-keys" Oct 31 02:11:28.229124 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.229102 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.330632 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.330604 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.330817 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.330661 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkng\" (UniqueName: \"kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.330817 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.330785 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.331714 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.331685 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:11:28.331804 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.331749 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:11:28.334195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.334174 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q"] Oct 31 02:11:28.432427 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.432392 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkng\" (UniqueName: \"kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.432635 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.432517 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.432635 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.432630 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.433493 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.433470 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.435842 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.435818 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.489709 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.489635 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkng\" (UniqueName: \"kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng\") pod \"maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.542297 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.542258 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:28.924985 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:28.924959 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q"] Oct 31 02:11:28.927198 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:11:28.927170 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad469d8_20e4_4d03_ad35_27430562fe18.slice/crio-9682eedc9718b6403075ac541851abb763e890a0cb4d61f678318ac8fc0e10d5 WatchSource:0}: Error finding container 9682eedc9718b6403075ac541851abb763e890a0cb4d61f678318ac8fc0e10d5: Status 404 returned error can't find the container with id 9682eedc9718b6403075ac541851abb763e890a0cb4d61f678318ac8fc0e10d5 Oct 31 02:11:29.238663 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:29.238630 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" event={"ID":"fad469d8-20e4-4d03-ad35-27430562fe18","Type":"ContainerStarted","Data":"9682eedc9718b6403075ac541851abb763e890a0cb4d61f678318ac8fc0e10d5"} Oct 31 02:11:29.372098 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:29.372064 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx"] Oct 31 02:11:29.384093 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:29.384062 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ec695b-4hgkx"] Oct 31 02:11:30.247261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:30.247174 5756 generic.go:356] "Generic (PLEG): container finished" podID="fad469d8-20e4-4d03-ad35-27430562fe18" containerID="d3cb2d40932ce92372652ee37cef31709717d257102bc6d7c653ebe77465a7d4" exitCode=0 Oct 31 02:11:30.247644 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:30.247259 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" event={"ID":"fad469d8-20e4-4d03-ad35-27430562fe18","Type":"ContainerDied","Data":"d3cb2d40932ce92372652ee37cef31709717d257102bc6d7c653ebe77465a7d4"} Oct 31 02:11:30.310092 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:30.310062 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1f5081-3210-4db3-b491-f30c23b8dbcf" path="/var/lib/kubelet/pods/3a1f5081-3210-4db3-b491-f30c23b8dbcf/volumes" Oct 31 02:11:30.611276 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:30.611187 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5"] Oct 31 02:11:30.613588 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:30.613569 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e621db-wxmt5"] Oct 31 02:11:31.396182 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.396159 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:31.566600 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.566565 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng\" (UniqueName: \"kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng\") pod \"fad469d8-20e4-4d03-ad35-27430562fe18\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " Oct 31 02:11:31.566797 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.566703 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca\") pod \"fad469d8-20e4-4d03-ad35-27430562fe18\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " Oct 31 02:11:31.566797 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.566757 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca\") pod \"fad469d8-20e4-4d03-ad35-27430562fe18\" (UID: \"fad469d8-20e4-4d03-ad35-27430562fe18\") " Oct 31 02:11:31.567085 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.567063 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca" pod "fad469d8-20e4-4d03-ad35-27430562fe18" (UID: "fad469d8-20e4-4d03-ad35-27430562fe18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:11:31.567223 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.567208 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:31.568905 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.568880 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca" pod "fad469d8-20e4-4d03-ad35-27430562fe18" (UID: "fad469d8-20e4-4d03-ad35-27430562fe18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:11:31.568905 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.568895 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng" pod "fad469d8-20e4-4d03-ad35-27430562fe18" (UID: "fad469d8-20e4-4d03-ad35-27430562fe18"). InnerVolumeSpecName "kube-api-access-mgkng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:11:31.667923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.667891 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fad469d8-20e4-4d03-ad35-27430562fe18-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:31.667923 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:31.667919 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkng\" (UniqueName: \"kubernetes.io/projected/fad469d8-20e4-4d03-ad35-27430562fe18-kube-api-access-mgkng\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:32.066573 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:11:32.066538 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:11:32.269663 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:32.269628 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" Oct 31 02:11:32.269663 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:32.269642 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q" event={"ID":"fad469d8-20e4-4d03-ad35-27430562fe18","Type":"ContainerDied","Data":"9682eedc9718b6403075ac541851abb763e890a0cb4d61f678318ac8fc0e10d5"} Oct 31 02:11:32.269947 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:32.269721 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9682eedc9718b6403075ac541851abb763e890a0cb4d61f678318ac8fc0e10d5" Oct 31 02:11:32.310327 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:32.310286 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184c0f44-1cf3-4d6b-a263-2d6c25abc7b8" path="/var/lib/kubelet/pods/184c0f44-1cf3-4d6b-a263-2d6c25abc7b8/volumes" Oct 31 02:11:39.628731 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:39.628692 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf"] Oct 31 02:11:39.733742 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:39.733711 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f"] Oct 31 02:11:44.629520 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:44.629457 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" podUID="b99d1ff5-a9fb-428b-bd03-8996f9dd743c" containerName="payload-processing" containerID="cri-o://a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817" gracePeriod=25 Oct 31 02:11:44.734253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:44.734190 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" podUID="77012e9c-e3d0-4e95-9506-99f2785a793e" containerName="payload-pre-processing" containerID="cri-o://a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9" gracePeriod=25 Oct 31 02:11:44.897190 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:44.897127 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:11:45.000482 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.000458 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:11:45.005446 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.005425 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume\") pod \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " Oct 31 02:11:45.005527 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.005468 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw\" (UniqueName: \"kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw\") pod \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\" (UID: \"b99d1ff5-a9fb-428b-bd03-8996f9dd743c\") " Oct 31 02:11:45.005845 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.005821 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume" pod "b99d1ff5-a9fb-428b-bd03-8996f9dd743c" (UID: "b99d1ff5-a9fb-428b-bd03-8996f9dd743c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:11:45.007602 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.007585 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw" pod "b99d1ff5-a9fb-428b-bd03-8996f9dd743c" (UID: "b99d1ff5-a9fb-428b-bd03-8996f9dd743c"). InnerVolumeSpecName "kube-api-access-nj2zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:11:45.106637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.106608 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z\" (UniqueName: \"kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z\") pod \"77012e9c-e3d0-4e95-9506-99f2785a793e\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " Oct 31 02:11:45.106801 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.106758 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume\") pod \"77012e9c-e3d0-4e95-9506-99f2785a793e\" (UID: \"77012e9c-e3d0-4e95-9506-99f2785a793e\") " Oct 31 02:11:45.107035 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.107018 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:45.107126 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.107036 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj2zw\" (UniqueName: \"kubernetes.io/projected/b99d1ff5-a9fb-428b-bd03-8996f9dd743c-kube-api-access-nj2zw\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:45.107168 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.107122 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume" pod "77012e9c-e3d0-4e95-9506-99f2785a793e" (UID: "77012e9c-e3d0-4e95-9506-99f2785a793e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:11:45.108674 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.108651 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z" pod "77012e9c-e3d0-4e95-9506-99f2785a793e" (UID: "77012e9c-e3d0-4e95-9506-99f2785a793e"). InnerVolumeSpecName "kube-api-access-ktb7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:11:45.208161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.208061 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77012e9c-e3d0-4e95-9506-99f2785a793e-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:45.208161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.208101 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktb7z\" (UniqueName: \"kubernetes.io/projected/77012e9c-e3d0-4e95-9506-99f2785a793e-kube-api-access-ktb7z\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:11:45.350765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.350736 5756 generic.go:356] "Generic (PLEG): container finished" podID="77012e9c-e3d0-4e95-9506-99f2785a793e" containerID="a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9" exitCode=0 Oct 31 02:11:45.350988 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.350786 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" Oct 31 02:11:45.350988 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.350816 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" event={"ID":"77012e9c-e3d0-4e95-9506-99f2785a793e","Type":"ContainerDied","Data":"a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9"} Oct 31 02:11:45.350988 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.350882 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f" event={"ID":"77012e9c-e3d0-4e95-9506-99f2785a793e","Type":"ContainerDied","Data":"e55afa2c00d49a40d51adc998854a3875914bb40438118acdcab85f3e1ad7ec1"} Oct 31 02:11:45.350988 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.350907 5756 scope.go:122] "RemoveContainer" containerID="a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9" Oct 31 02:11:45.352416 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.352389 5756 generic.go:356] "Generic (PLEG): container finished" podID="b99d1ff5-a9fb-428b-bd03-8996f9dd743c" containerID="a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817" exitCode=0 Oct 31 02:11:45.352528 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.352478 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" event={"ID":"b99d1ff5-a9fb-428b-bd03-8996f9dd743c","Type":"ContainerDied","Data":"a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817"} Oct 31 02:11:45.352528 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.352518 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" event={"ID":"b99d1ff5-a9fb-428b-bd03-8996f9dd743c","Type":"ContainerDied","Data":"db3ed885f3623e78b7511b96bb5b8bdae4dfd42928a22e7aeadd590ee958c06a"} Oct 31 02:11:45.352617 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.352606 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf" Oct 31 02:11:45.362967 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.362943 5756 scope.go:122] "RemoveContainer" containerID="a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9" Oct 31 02:11:45.363213 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:11:45.363194 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9\": container with ID starting with a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9 not found: ID does not exist" containerID="a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9" Oct 31 02:11:45.363301 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.363226 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9"} err="failed to get container status \"a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9\": rpc error: code = NotFound desc = could not find container \"a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9\": container with ID starting with a502ef030ba650c2d784a63b3c7016f1292d7efcd61af160fd2020170bb35df9 not found: ID does not exist" Oct 31 02:11:45.363301 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.363246 5756 scope.go:122] "RemoveContainer" containerID="a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817" Oct 31 02:11:45.372138 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.372118 5756 scope.go:122] "RemoveContainer" containerID="a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817" Oct 31 02:11:45.372425 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:11:45.372405 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817\": container with ID starting with a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817 not found: ID does not exist" containerID="a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817" Oct 31 02:11:45.372476 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.372433 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817"} err="failed to get container status \"a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817\": rpc error: code = NotFound desc = could not find container \"a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817\": container with ID starting with a8dd5663e999ec7ac6be49950404b12629a5eb68070734244341bf0322560817 not found: ID does not exist" Oct 31 02:11:45.512703 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.512623 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f"] Oct 31 02:11:45.516047 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.516023 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a713e6-858ff99cc5-nl77f"] Oct 31 02:11:45.623171 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.623140 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf"] Oct 31 02:11:45.627199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:45.627171 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a713e6-57b7d787df-xlrvf"] Oct 31 02:11:46.311197 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:46.311165 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77012e9c-e3d0-4e95-9506-99f2785a793e" path="/var/lib/kubelet/pods/77012e9c-e3d0-4e95-9506-99f2785a793e/volumes" Oct 31 02:11:46.311961 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:11:46.311742 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99d1ff5-a9fb-428b-bd03-8996f9dd743c" path="/var/lib/kubelet/pods/b99d1ff5-a9fb-428b-bd03-8996f9dd743c/volumes" Oct 31 02:12:00.825269 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:00.825219 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q"] Oct 31 02:12:00.874412 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:00.874382 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a713e6-t6t9q"] Oct 31 02:12:02.066410 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:12:02.066373 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:12:02.310732 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:02.310695 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad469d8-20e4-4d03-ad35-27430562fe18" path="/var/lib/kubelet/pods/fad469d8-20e4-4d03-ad35-27430562fe18/volumes" Oct 31 02:12:04.651483 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:04.651450 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:12:11.611375 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:11.611340 5756 scope.go:122] "RemoveContainer" containerID="9c185a400f29607a1f0621aa5cf483a67ffa44c0ea00e73ff9a806eb494179f0" Oct 31 02:12:11.620959 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:11.620937 5756 scope.go:122] "RemoveContainer" containerID="e9be40d357151bc377ea52af7d7a964a7c21a56e709acdea59296ba18a1a7c35" Oct 31 02:12:11.630684 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:11.630599 5756 scope.go:122] "RemoveContainer" containerID="76fe6cf284e2d6e7ef3c6da00927816daae95f7f2dfdb717ee44a73b6dbc15a9" Oct 31 02:12:17.720883 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:17.720794 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22"] Oct 31 02:12:17.721265 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:17.721098 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" containerID="cri-o://f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5" gracePeriod=30 Oct 31 02:12:18.552095 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552057 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24"] Oct 31 02:12:18.552634 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552614 5756 cpu_manager.go:414] "containerMap: removing container" podUID="b99d1ff5-a9fb-428b-bd03-8996f9dd743c" containerName="payload-processing" Oct 31 02:12:18.552749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552635 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99d1ff5-a9fb-428b-bd03-8996f9dd743c" containerName="payload-processing" Oct 31 02:12:18.552749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552652 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fad469d8-20e4-4d03-ad35-27430562fe18" containerName="revoke-keys" Oct 31 02:12:18.552749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552660 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad469d8-20e4-4d03-ad35-27430562fe18" containerName="revoke-keys" Oct 31 02:12:18.552749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552685 5756 cpu_manager.go:414] "containerMap: removing container" podUID="77012e9c-e3d0-4e95-9506-99f2785a793e" containerName="payload-pre-processing" Oct 31 02:12:18.552749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552692 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77012e9c-e3d0-4e95-9506-99f2785a793e" containerName="payload-pre-processing" Oct 31 02:12:18.553054 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552851 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99d1ff5-a9fb-428b-bd03-8996f9dd743c" containerName="payload-processing" Oct 31 02:12:18.553054 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552883 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad469d8-20e4-4d03-ad35-27430562fe18" containerName="revoke-keys" Oct 31 02:12:18.553054 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.552894 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="77012e9c-e3d0-4e95-9506-99f2785a793e" containerName="payload-pre-processing" Oct 31 02:12:18.556292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.556269 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.640213 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.640179 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.640402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.640380 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.640464 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.640424 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztmk2\" (UniqueName: \"kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.658573 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.658548 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:12:18.658573 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.658554 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:12:18.660682 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.660661 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24"] Oct 31 02:12:18.741449 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.741419 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.741827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.741455 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztmk2\" (UniqueName: \"kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.741827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.741578 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.742257 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.742240 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.743895 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.743855 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.797643 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.797616 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztmk2\" (UniqueName: \"kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2\") pod \"maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:18.868798 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:18.868725 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:19.239729 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:19.239688 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24"] Oct 31 02:12:19.242194 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:12:19.242158 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbceaac_6b2f_4344_b968_998e4776dfde.slice/crio-0cc070f04d943d935d2d2e7ab921079507e7777f873c10955004e900e86f811c WatchSource:0}: Error finding container 0cc070f04d943d935d2d2e7ab921079507e7777f873c10955004e900e86f811c: Status 404 returned error can't find the container with id 0cc070f04d943d935d2d2e7ab921079507e7777f873c10955004e900e86f811c Oct 31 02:12:19.567454 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:19.567417 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" event={"ID":"3fbceaac-6b2f-4344-b968-998e4776dfde","Type":"ContainerStarted","Data":"8fd7f6c515c7720145078a0fbe03f99534292602d45e452b809f2a723aaf49f8"} Oct 31 02:12:19.567623 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:19.567473 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" event={"ID":"3fbceaac-6b2f-4344-b968-998e4776dfde","Type":"ContainerStarted","Data":"0cc070f04d943d935d2d2e7ab921079507e7777f873c10955004e900e86f811c"} Oct 31 02:12:19.680453 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:19.680398 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" podStartSLOduration=1.6803838359999999 podStartE2EDuration="1.680383836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:19.679062302 +0000 UTC m=+4511.486029870" watchObservedRunningTime="2026-10-31 02:12:19.680383836 +0000 UTC m=+4511.487351379" Oct 31 02:12:20.575518 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:20.575484 5756 generic.go:356] "Generic (PLEG): container finished" podID="3fbceaac-6b2f-4344-b968-998e4776dfde" containerID="8fd7f6c515c7720145078a0fbe03f99534292602d45e452b809f2a723aaf49f8" exitCode=0 Oct 31 02:12:20.576001 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:20.575564 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" event={"ID":"3fbceaac-6b2f-4344-b968-998e4776dfde","Type":"ContainerDied","Data":"8fd7f6c515c7720145078a0fbe03f99534292602d45e452b809f2a723aaf49f8"} Oct 31 02:12:21.719674 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.719651 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:21.771169 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.771140 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca\") pod \"3fbceaac-6b2f-4344-b968-998e4776dfde\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " Oct 31 02:12:21.771350 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.771221 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2\" (UniqueName: \"kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2\") pod \"3fbceaac-6b2f-4344-b968-998e4776dfde\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " Oct 31 02:12:21.771350 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.771283 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca\") pod \"3fbceaac-6b2f-4344-b968-998e4776dfde\" (UID: \"3fbceaac-6b2f-4344-b968-998e4776dfde\") " Oct 31 02:12:21.771537 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.771513 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca" pod "3fbceaac-6b2f-4344-b968-998e4776dfde" (UID: "3fbceaac-6b2f-4344-b968-998e4776dfde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:21.771622 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.771598 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:21.773370 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.773342 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2" pod "3fbceaac-6b2f-4344-b968-998e4776dfde" (UID: "3fbceaac-6b2f-4344-b968-998e4776dfde"). InnerVolumeSpecName "kube-api-access-ztmk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:21.773487 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.773389 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca" pod "3fbceaac-6b2f-4344-b968-998e4776dfde" (UID: "3fbceaac-6b2f-4344-b968-998e4776dfde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:12:21.873197 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.873124 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztmk2\" (UniqueName: \"kubernetes.io/projected/3fbceaac-6b2f-4344-b968-998e4776dfde-kube-api-access-ztmk2\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:21.873197 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:21.873151 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbceaac-6b2f-4344-b968-998e4776dfde-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:22.588458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:22.588370 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" Oct 31 02:12:22.588458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:22.588400 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24" event={"ID":"3fbceaac-6b2f-4344-b968-998e4776dfde","Type":"ContainerDied","Data":"0cc070f04d943d935d2d2e7ab921079507e7777f873c10955004e900e86f811c"} Oct 31 02:12:22.588458 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:22.588444 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cc070f04d943d935d2d2e7ab921079507e7777f873c10955004e900e86f811c" Oct 31 02:12:22.982171 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:22.982147 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:12:23.085717 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.085683 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.085717 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.085722 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.085745 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.085776 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.085924 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086026 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.085987 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086264 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086062 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086264 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086111 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086264 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086187 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs\") pod \"106248d9-94d9-4c0e-8dba-07fe9718d991\" (UID: \"106248d9-94d9-4c0e-8dba-07fe9718d991\") " Oct 31 02:12:23.086438 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086108 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:12:23.086596 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086555 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.086690 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086611 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:12:23.086973 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.086762 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:23.087084 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.087059 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:12:23.087239 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.087214 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:12:23.088424 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.088398 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:23.088424 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.088410 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:12:23.088572 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.088557 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "kube-api-access-pnrv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:23.088698 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.088683 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo" pod "106248d9-94d9-4c0e-8dba-07fe9718d991" (UID: "106248d9-94d9-4c0e-8dba-07fe9718d991"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:12:23.187202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187174 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187200 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/106248d9-94d9-4c0e-8dba-07fe9718d991-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187202 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187210 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187444 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187220 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187444 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187229 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/106248d9-94d9-4c0e-8dba-07fe9718d991-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187444 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187238 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnrv4\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-kube-api-access-pnrv4\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187444 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187248 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.187444 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.187256 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/106248d9-94d9-4c0e-8dba-07fe9718d991-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:23.596503 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.596471 5756 generic.go:356] "Generic (PLEG): container finished" podID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerID="f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5" exitCode=0 Oct 31 02:12:23.596726 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.596522 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" Oct 31 02:12:23.596726 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.596552 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" event={"ID":"106248d9-94d9-4c0e-8dba-07fe9718d991","Type":"ContainerDied","Data":"f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5"} Oct 31 02:12:23.596726 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.596615 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22" event={"ID":"106248d9-94d9-4c0e-8dba-07fe9718d991","Type":"ContainerDied","Data":"a01fef8d228dfd791972b989e0f60b9dc3bf594ac5c651f6a97a03a93d52d361"} Oct 31 02:12:23.596726 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.596649 5756 scope.go:122] "RemoveContainer" containerID="f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5" Oct 31 02:12:23.610347 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.610326 5756 scope.go:122] "RemoveContainer" containerID="8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d" Oct 31 02:12:23.619734 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.619715 5756 scope.go:122] "RemoveContainer" containerID="f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5" Oct 31 02:12:23.620082 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:12:23.620058 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5\": container with ID starting with f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5 not found: ID does not exist" containerID="f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5" Oct 31 02:12:23.620137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.620093 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5"} err="failed to get container status \"f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5\": rpc error: code = NotFound desc = could not find container \"f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5\": container with ID starting with f78ff348a7dca2af2f0d01d6739dbbf776591addc4600a4dd66cfb38f394ccf5 not found: ID does not exist" Oct 31 02:12:23.620137 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.620115 5756 scope.go:122] "RemoveContainer" containerID="8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d" Oct 31 02:12:23.620360 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:12:23.620343 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d\": container with ID starting with 8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d not found: ID does not exist" containerID="8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d" Oct 31 02:12:23.620420 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.620370 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d"} err="failed to get container status \"8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d\": rpc error: code = NotFound desc = could not find container \"8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d\": container with ID starting with 8f50601007575262787b5c01a81bf87d50078f1c24feedd4a67339f670927f2d not found: ID does not exist" Oct 31 02:12:23.760426 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.760393 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22"] Oct 31 02:12:23.810463 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:23.810433 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a713e6-openshift-default-686545875c-59f22"] Oct 31 02:12:24.310750 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:24.310718 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" path="/var/lib/kubelet/pods/106248d9-94d9-4c0e-8dba-07fe9718d991/volumes" Oct 31 02:12:30.032957 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:30.032914 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7"] Oct 31 02:12:30.142317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:30.142281 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl"] Oct 31 02:12:32.066320 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:12:32.066283 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:12:35.034063 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.033992 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" podUID="6afc27c5-e168-4efb-9fb1-d77fed08594d" containerName="payload-processing" containerID="cri-o://d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f" gracePeriod=25 Oct 31 02:12:35.143308 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.143264 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" podUID="0859dae0-de47-43e0-9aff-2d1caaae9760" containerName="payload-pre-processing" containerID="cri-o://d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209" gracePeriod=25 Oct 31 02:12:35.274833 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.274804 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:12:35.397429 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.397403 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:12:35.416341 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.416310 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume\") pod \"6afc27c5-e168-4efb-9fb1-d77fed08594d\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " Oct 31 02:12:35.416490 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.416407 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc\" (UniqueName: \"kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc\") pod \"6afc27c5-e168-4efb-9fb1-d77fed08594d\" (UID: \"6afc27c5-e168-4efb-9fb1-d77fed08594d\") " Oct 31 02:12:35.416675 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.416640 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume" pod "6afc27c5-e168-4efb-9fb1-d77fed08594d" (UID: "6afc27c5-e168-4efb-9fb1-d77fed08594d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:35.416919 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.416898 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6afc27c5-e168-4efb-9fb1-d77fed08594d-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:35.418624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.418600 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc" pod "6afc27c5-e168-4efb-9fb1-d77fed08594d" (UID: "6afc27c5-e168-4efb-9fb1-d77fed08594d"). InnerVolumeSpecName "kube-api-access-vzrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:35.523972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.523226 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume\") pod \"0859dae0-de47-43e0-9aff-2d1caaae9760\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " Oct 31 02:12:35.523972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.523284 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp\" (UniqueName: \"kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp\") pod \"0859dae0-de47-43e0-9aff-2d1caaae9760\" (UID: \"0859dae0-de47-43e0-9aff-2d1caaae9760\") " Oct 31 02:12:35.523972 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.523602 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrxc\" (UniqueName: \"kubernetes.io/projected/6afc27c5-e168-4efb-9fb1-d77fed08594d-kube-api-access-vzrxc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:35.540113 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.540043 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume" pod "0859dae0-de47-43e0-9aff-2d1caaae9760" (UID: "0859dae0-de47-43e0-9aff-2d1caaae9760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:35.541975 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.541933 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp" pod "0859dae0-de47-43e0-9aff-2d1caaae9760" (UID: "0859dae0-de47-43e0-9aff-2d1caaae9760"). InnerVolumeSpecName "kube-api-access-f6gmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:35.624925 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.624792 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0859dae0-de47-43e0-9aff-2d1caaae9760-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:35.624925 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.624824 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gmp\" (UniqueName: \"kubernetes.io/projected/0859dae0-de47-43e0-9aff-2d1caaae9760-kube-api-access-f6gmp\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:12:35.675071 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.675033 5756 generic.go:356] "Generic (PLEG): container finished" podID="0859dae0-de47-43e0-9aff-2d1caaae9760" containerID="d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209" exitCode=0 Oct 31 02:12:35.675280 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.675084 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" Oct 31 02:12:35.675280 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.675117 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" event={"ID":"0859dae0-de47-43e0-9aff-2d1caaae9760","Type":"ContainerDied","Data":"d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209"} Oct 31 02:12:35.675280 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.675180 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl" event={"ID":"0859dae0-de47-43e0-9aff-2d1caaae9760","Type":"ContainerDied","Data":"04ef695b0ede6ba647ad6f30e5db7acfcbcf2d85270ce164ba363016a6ea2fe5"} Oct 31 02:12:35.675280 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.675212 5756 scope.go:122] "RemoveContainer" containerID="d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209" Oct 31 02:12:35.676756 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.676733 5756 generic.go:356] "Generic (PLEG): container finished" podID="6afc27c5-e168-4efb-9fb1-d77fed08594d" containerID="d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f" exitCode=0 Oct 31 02:12:35.676856 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.676783 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" Oct 31 02:12:35.676856 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.676819 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" event={"ID":"6afc27c5-e168-4efb-9fb1-d77fed08594d","Type":"ContainerDied","Data":"d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f"} Oct 31 02:12:35.676981 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.676890 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7" event={"ID":"6afc27c5-e168-4efb-9fb1-d77fed08594d","Type":"ContainerDied","Data":"31fb00a1c19126147281f3260bacaaf5e6a83ef7c370f0ceb0aa37101c1fdd6c"} Oct 31 02:12:35.686750 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.686734 5756 scope.go:122] "RemoveContainer" containerID="d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209" Oct 31 02:12:35.687045 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:12:35.687026 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209\": container with ID starting with d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209 not found: ID does not exist" containerID="d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209" Oct 31 02:12:35.687148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.687056 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209"} err="failed to get container status \"d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209\": rpc error: code = NotFound desc = could not find container \"d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209\": container with ID starting with d09fa986c35908bd8eb966d0f6431b0ab1f0573f31259ce277573e97a3896209 not found: ID does not exist" Oct 31 02:12:35.687148 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.687078 5756 scope.go:122] "RemoveContainer" containerID="d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f" Oct 31 02:12:35.695826 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.695810 5756 scope.go:122] "RemoveContainer" containerID="d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f" Oct 31 02:12:35.696122 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:12:35.696102 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f\": container with ID starting with d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f not found: ID does not exist" containerID="d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f" Oct 31 02:12:35.696195 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.696134 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f"} err="failed to get container status \"d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f\": rpc error: code = NotFound desc = could not find container \"d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f\": container with ID starting with d250d77a44c5d13140cfa587266e30845ccbe4b604b92560085e14108dbddf2f not found: ID does not exist" Oct 31 02:12:35.838899 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.838851 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7"] Oct 31 02:12:35.847027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.847002 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b91499-697b648765-58gq7"] Oct 31 02:12:35.954291 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.954209 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl"] Oct 31 02:12:35.957650 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:35.957622 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b91499-7b94874f5f-92kzl"] Oct 31 02:12:36.310274 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:36.310244 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0859dae0-de47-43e0-9aff-2d1caaae9760" path="/var/lib/kubelet/pods/0859dae0-de47-43e0-9aff-2d1caaae9760/volumes" Oct 31 02:12:36.310650 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:36.310617 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afc27c5-e168-4efb-9fb1-d77fed08594d" path="/var/lib/kubelet/pods/6afc27c5-e168-4efb-9fb1-d77fed08594d/volumes" Oct 31 02:12:54.497946 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:54.497914 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24"] Oct 31 02:12:54.546877 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:54.546836 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b91499-w2r24"] Oct 31 02:12:56.310783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:56.310748 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbceaac-6b2f-4344-b968-998e4776dfde" path="/var/lib/kubelet/pods/3fbceaac-6b2f-4344-b968-998e4776dfde/volumes" Oct 31 02:12:57.948260 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:12:57.948223 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:13:02.067989 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:13:02.067950 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:13:02.322436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322358 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn"] Oct 31 02:13:02.322791 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322778 5756 cpu_manager.go:414] "containerMap: removing container" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" Oct 31 02:13:02.322838 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322792 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" Oct 31 02:13:02.322838 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322826 5756 cpu_manager.go:414] "containerMap: removing container" podUID="6afc27c5-e168-4efb-9fb1-d77fed08594d" containerName="payload-processing" Oct 31 02:13:02.322838 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322831 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afc27c5-e168-4efb-9fb1-d77fed08594d" containerName="payload-processing" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322842 5756 cpu_manager.go:414] "containerMap: removing container" podUID="0859dae0-de47-43e0-9aff-2d1caaae9760" containerName="payload-pre-processing" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322848 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0859dae0-de47-43e0-9aff-2d1caaae9760" containerName="payload-pre-processing" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322857 5756 cpu_manager.go:414] "containerMap: removing container" podUID="3fbceaac-6b2f-4344-b968-998e4776dfde" containerName="revoke-keys" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322877 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbceaac-6b2f-4344-b968-998e4776dfde" containerName="revoke-keys" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322943 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322954 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" Oct 31 02:13:02.322960 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322962 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afc27c5-e168-4efb-9fb1-d77fed08594d" containerName="payload-processing" Oct 31 02:13:02.323187 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322968 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="0859dae0-de47-43e0-9aff-2d1caaae9760" containerName="payload-pre-processing" Oct 31 02:13:02.323187 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.322974 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbceaac-6b2f-4344-b968-998e4776dfde" containerName="revoke-keys" Oct 31 02:13:02.326188 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.326173 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.382246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.382212 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn"] Oct 31 02:13:02.396259 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.396228 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4q6\" (UniqueName: \"kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.396404 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.396358 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.396480 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.396430 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.428759 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.428733 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:13:02.428937 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.428808 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:13:02.497696 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.497660 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.497930 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.497705 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.497930 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.497788 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4q6\" (UniqueName: \"kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.498419 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.498394 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.500284 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.500264 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.553724 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.553692 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4q6\" (UniqueName: \"kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:02.639632 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:02.639553 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:03.034279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:03.034254 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn"] Oct 31 02:13:03.036398 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:13:03.036365 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84694f57_8005_477a_b6a2_e5853f06b072.slice/crio-17dfd4446334c67ac4f1e16a3beebcf4ca283c2c1ced4c977429d44333d3983d WatchSource:0}: Error finding container 17dfd4446334c67ac4f1e16a3beebcf4ca283c2c1ced4c977429d44333d3983d: Status 404 returned error can't find the container with id 17dfd4446334c67ac4f1e16a3beebcf4ca283c2c1ced4c977429d44333d3983d Oct 31 02:13:03.855832 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:03.855799 5756 generic.go:356] "Generic (PLEG): container finished" podID="84694f57-8005-477a-b6a2-e5853f06b072" containerID="9daa875b28d3c4cf3cbe19c9c3ac374b7d5ebab697dfb01de7e76c725b2bb433" exitCode=0 Oct 31 02:13:03.856308 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:03.855901 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" event={"ID":"84694f57-8005-477a-b6a2-e5853f06b072","Type":"ContainerDied","Data":"9daa875b28d3c4cf3cbe19c9c3ac374b7d5ebab697dfb01de7e76c725b2bb433"} Oct 31 02:13:03.856308 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:03.855963 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" event={"ID":"84694f57-8005-477a-b6a2-e5853f06b072","Type":"ContainerStarted","Data":"17dfd4446334c67ac4f1e16a3beebcf4ca283c2c1ced4c977429d44333d3983d"} Oct 31 02:13:05.000102 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.000074 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:05.122827 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.122754 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6\" (UniqueName: \"kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6\") pod \"84694f57-8005-477a-b6a2-e5853f06b072\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " Oct 31 02:13:05.123016 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.122826 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca\") pod \"84694f57-8005-477a-b6a2-e5853f06b072\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " Oct 31 02:13:05.123016 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.122854 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca\") pod \"84694f57-8005-477a-b6a2-e5853f06b072\" (UID: \"84694f57-8005-477a-b6a2-e5853f06b072\") " Oct 31 02:13:05.123274 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.123248 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca" pod "84694f57-8005-477a-b6a2-e5853f06b072" (UID: "84694f57-8005-477a-b6a2-e5853f06b072"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:05.125076 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.125048 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6" pod "84694f57-8005-477a-b6a2-e5853f06b072" (UID: "84694f57-8005-477a-b6a2-e5853f06b072"). InnerVolumeSpecName "kube-api-access-sv4q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:05.125076 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.125070 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca" pod "84694f57-8005-477a-b6a2-e5853f06b072" (UID: "84694f57-8005-477a-b6a2-e5853f06b072"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:05.223931 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.223890 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4q6\" (UniqueName: \"kubernetes.io/projected/84694f57-8005-477a-b6a2-e5853f06b072-kube-api-access-sv4q6\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:05.223931 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.223922 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:05.224118 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.223944 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84694f57-8005-477a-b6a2-e5853f06b072-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:05.869271 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.869237 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:05.869271 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.869256 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" event={"ID":"84694f57-8005-477a-b6a2-e5853f06b072","Type":"ContainerDied","Data":"17dfd4446334c67ac4f1e16a3beebcf4ca283c2c1ced4c977429d44333d3983d"} Oct 31 02:13:05.869492 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:05.869297 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17dfd4446334c67ac4f1e16a3beebcf4ca283c2c1ced4c977429d44333d3983d" Oct 31 02:13:11.673221 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:11.673186 5756 scope.go:122] "RemoveContainer" containerID="1003daa7d4a002a03460dbabca312707aa86ffd5ee9b25e9692af55377c2762e" Oct 31 02:13:11.683024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:11.683003 5756 scope.go:122] "RemoveContainer" containerID="88615ef2682f35f8b29b127cbd306086b28e941a26e8016f5d6daa832fd4e6b0" Oct 31 02:13:11.691770 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:11.691743 5756 scope.go:122] "RemoveContainer" containerID="41dbe3e2d32fe00e164e4e977375796405f7f132b95669eeb087fefddd41b95f" Oct 31 02:13:13.501653 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:13.501619 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp"] Oct 31 02:13:14.051068 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:14.051034 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg"] Oct 31 02:13:14.055036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:14.055005 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc"] Oct 31 02:13:14.100581 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:14.100547 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c422736-mf2fg"] Oct 31 02:13:14.102335 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:14.102314 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0966fd06-bhhsc"] Oct 31 02:13:14.310456 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:14.310381 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf347bbd-c700-41b4-b5c1-44f637bffab8" path="/var/lib/kubelet/pods/bf347bbd-c700-41b4-b5c1-44f637bffab8/volumes" Oct 31 02:13:14.310788 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:14.310775 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9eff32c-e918-4a22-ade4-0ff28696fff4" path="/var/lib/kubelet/pods/c9eff32c-e918-4a22-ade4-0ff28696fff4/volumes" Oct 31 02:13:18.262961 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:18.262928 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn"] Oct 31 02:13:18.312120 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:18.312084 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn"] Oct 31 02:13:18.357969 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:13:18.357931 5756 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn\" is forbidden: User \"system:node:ip-10-0-136-249.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-136-249.us-west-1.compute.internal' and this object" podUID="84694f57-8005-477a-b6a2-e5853f06b072" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b6f24b-bxkfn" Oct 31 02:13:18.476109 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:18.476076 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:13:18.502662 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:18.502630 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerName="payload-processing" containerID="cri-o://312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9" gracePeriod=25 Oct 31 02:13:20.310691 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:20.310659 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84694f57-8005-477a-b6a2-e5853f06b072" path="/var/lib/kubelet/pods/84694f57-8005-477a-b6a2-e5853f06b072/volumes" Oct 31 02:13:24.847894 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:24.847827 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b"] Oct 31 02:13:24.848403 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:24.848082 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" containerID="cri-o://f934569aae395a604139df8d0348336ab4e41cbaa8e21b8da8c3a7d8f2cfd35c" gracePeriod=30 Oct 31 02:13:25.756990 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.756952 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m"] Oct 31 02:13:25.757408 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.757395 5756 cpu_manager.go:414] "containerMap: removing container" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" Oct 31 02:13:25.757453 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.757409 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106248d9-94d9-4c0e-8dba-07fe9718d991" containerName="istio-proxy" Oct 31 02:13:25.757453 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.757423 5756 cpu_manager.go:414] "containerMap: removing container" podUID="84694f57-8005-477a-b6a2-e5853f06b072" containerName="revoke-keys" Oct 31 02:13:25.757453 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.757428 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84694f57-8005-477a-b6a2-e5853f06b072" containerName="revoke-keys" Oct 31 02:13:25.757561 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.757503 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="84694f57-8005-477a-b6a2-e5853f06b072" containerName="revoke-keys" Oct 31 02:13:25.760821 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.760797 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.814828 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.814795 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:13:25.815016 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.814981 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:13:25.817011 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.816987 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m"] Oct 31 02:13:25.829785 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.829753 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4wx\" (UniqueName: \"kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.829955 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.829882 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.830078 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.830058 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.931290 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.931251 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.931705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.931339 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4wx\" (UniqueName: \"kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.931705 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.931440 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.932199 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.932178 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.933812 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.933782 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:25.988174 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:25.988143 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4wx\" (UniqueName: \"kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:26.074106 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:26.074071 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:26.178354 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:26.178319 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:26.446035 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:26.446010 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m"] Oct 31 02:13:26.448540 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:13:26.448513 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce6ea27_e7b9_44c1_8674_67d8f30f575a.slice/crio-457cd330b941000110577c3e2b2bc39978f9e761311fa7e4891fbb7c57dfa757 WatchSource:0}: Error finding container 457cd330b941000110577c3e2b2bc39978f9e761311fa7e4891fbb7c57dfa757: Status 404 returned error can't find the container with id 457cd330b941000110577c3e2b2bc39978f9e761311fa7e4891fbb7c57dfa757 Oct 31 02:13:26.996760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:26.996673 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" event={"ID":"fce6ea27-e7b9-44c1-8674-67d8f30f575a","Type":"ContainerStarted","Data":"3b217f7a6332c7565628c4343ed4df879f9fdfea0534e5c21838b14fdee4ac57"} Oct 31 02:13:26.996760 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:26.996727 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" event={"ID":"fce6ea27-e7b9-44c1-8674-67d8f30f575a","Type":"ContainerStarted","Data":"457cd330b941000110577c3e2b2bc39978f9e761311fa7e4891fbb7c57dfa757"} Oct 31 02:13:27.109919 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:27.109850 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" podStartSLOduration=2.10983571 podStartE2EDuration="2.10983571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:27.108243611 +0000 UTC m=+4578.915211168" watchObservedRunningTime="2026-10-31 02:13:27.10983571 +0000 UTC m=+4578.916803253" Oct 31 02:13:28.009024 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:28.008989 5756 generic.go:356] "Generic (PLEG): container finished" podID="fce6ea27-e7b9-44c1-8674-67d8f30f575a" containerID="3b217f7a6332c7565628c4343ed4df879f9fdfea0534e5c21838b14fdee4ac57" exitCode=0 Oct 31 02:13:28.009499 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:28.009079 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" event={"ID":"fce6ea27-e7b9-44c1-8674-67d8f30f575a","Type":"ContainerDied","Data":"3b217f7a6332c7565628c4343ed4df879f9fdfea0534e5c21838b14fdee4ac57"} Oct 31 02:13:29.155534 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.155510 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:29.158826 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.158807 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx\" (UniqueName: \"kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx\") pod \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " Oct 31 02:13:29.158933 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.158855 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca\") pod \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " Oct 31 02:13:29.158998 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.158952 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca\") pod \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\" (UID: \"fce6ea27-e7b9-44c1-8674-67d8f30f575a\") " Oct 31 02:13:29.159334 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.159312 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca" pod "fce6ea27-e7b9-44c1-8674-67d8f30f575a" (UID: "fce6ea27-e7b9-44c1-8674-67d8f30f575a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:29.161033 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.161006 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx" pod "fce6ea27-e7b9-44c1-8674-67d8f30f575a" (UID: "fce6ea27-e7b9-44c1-8674-67d8f30f575a"). InnerVolumeSpecName "kube-api-access-qx4wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:29.161127 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.161027 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca" pod "fce6ea27-e7b9-44c1-8674-67d8f30f575a" (UID: "fce6ea27-e7b9-44c1-8674-67d8f30f575a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:29.260511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.260474 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:29.260511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.260505 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx4wx\" (UniqueName: \"kubernetes.io/projected/fce6ea27-e7b9-44c1-8674-67d8f30f575a-kube-api-access-qx4wx\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:29.260511 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.260517 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fce6ea27-e7b9-44c1-8674-67d8f30f575a-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:29.863052 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.863025 5756 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 31 02:13:29.863190 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:29.863075 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 31 02:13:30.024612 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.022848 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" Oct 31 02:13:30.024612 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.022893 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m" event={"ID":"fce6ea27-e7b9-44c1-8674-67d8f30f575a","Type":"ContainerDied","Data":"457cd330b941000110577c3e2b2bc39978f9e761311fa7e4891fbb7c57dfa757"} Oct 31 02:13:30.024612 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.022954 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="457cd330b941000110577c3e2b2bc39978f9e761311fa7e4891fbb7c57dfa757" Oct 31 02:13:30.026371 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.026112 5756 generic.go:356] "Generic (PLEG): container finished" podID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerID="f934569aae395a604139df8d0348336ab4e41cbaa8e21b8da8c3a7d8f2cfd35c" exitCode=0 Oct 31 02:13:30.026371 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.026167 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" event={"ID":"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0","Type":"ContainerDied","Data":"f934569aae395a604139df8d0348336ab4e41cbaa8e21b8da8c3a7d8f2cfd35c"} Oct 31 02:13:30.026371 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.026237 5756 scope.go:122] "RemoveContainer" containerID="4aa14ed803d05d5225ab160830f17c1cde122e021903bb6c9f86ed059ea00180" Oct 31 02:13:30.091376 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.091351 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 02:13:30.167407 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167372 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167426 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167448 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167479 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167502 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167535 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167600 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167649 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167689 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc\") pod \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\" (UID: \"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0\") " Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167704 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:30.167811 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167726 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.167836 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.168008 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.168112 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.168169 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.168191 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.168212 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.168627 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.168228 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.169910 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.169885 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:30.170007 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.169985 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:30.170078 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.170056 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:30.170125 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.170110 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc" pod "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" (UID: "f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0"). InnerVolumeSpecName "kube-api-access-85qbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:30.269261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.269213 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.269261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.269252 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.269261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.269261 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.269261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.269271 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:30.269261 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:30.269279 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85qbc\" (UniqueName: \"kubernetes.io/projected/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0-kube-api-access-85qbc\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:31.033146 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:31.033107 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" Oct 31 02:13:31.033146 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:31.033132 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b" event={"ID":"f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0","Type":"ContainerDied","Data":"51dee869a0c21932dec036d1a75c9810f13aec8d289c285cc818dad35c75e63a"} Oct 31 02:13:31.033396 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:31.033196 5756 scope.go:122] "RemoveContainer" containerID="f934569aae395a604139df8d0348336ab4e41cbaa8e21b8da8c3a7d8f2cfd35c" Oct 31 02:13:31.195644 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:31.195612 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b"] Oct 31 02:13:31.245205 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:31.245164 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b6f24b-openshift-default-7fbf48f695-nrs4b"] Oct 31 02:13:32.065930 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:13:32.065892 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:13:32.310402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:32.310372 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" path="/var/lib/kubelet/pods/f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0/volumes" Oct 31 02:13:36.177980 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:36.177948 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:37.230183 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:37.230141 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl"] Oct 31 02:13:38.766486 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.766420 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 02:13:38.852956 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.852918 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk\" (UniqueName: \"kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk\") pod \"298b64a8-a86b-4988-bfb5-bd0467920df2\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " Oct 31 02:13:38.853162 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.852989 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume\") pod \"298b64a8-a86b-4988-bfb5-bd0467920df2\" (UID: \"298b64a8-a86b-4988-bfb5-bd0467920df2\") " Oct 31 02:13:38.853519 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.853488 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume" pod "298b64a8-a86b-4988-bfb5-bd0467920df2" (UID: "298b64a8-a86b-4988-bfb5-bd0467920df2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:38.855364 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.855342 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk" pod "298b64a8-a86b-4988-bfb5-bd0467920df2" (UID: "298b64a8-a86b-4988-bfb5-bd0467920df2"). InnerVolumeSpecName "kube-api-access-htntk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:38.954581 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.954547 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htntk\" (UniqueName: \"kubernetes.io/projected/298b64a8-a86b-4988-bfb5-bd0467920df2-kube-api-access-htntk\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:38.954581 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:38.954579 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/298b64a8-a86b-4988-bfb5-bd0467920df2-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:39.082016 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.081979 5756 generic.go:356] "Generic (PLEG): container finished" podID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerID="312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9" exitCode=0 Oct 31 02:13:39.082253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.082023 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" Oct 31 02:13:39.082253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.082036 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" event={"ID":"298b64a8-a86b-4988-bfb5-bd0467920df2","Type":"ContainerDied","Data":"312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9"} Oct 31 02:13:39.082253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.082124 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp" event={"ID":"298b64a8-a86b-4988-bfb5-bd0467920df2","Type":"ContainerDied","Data":"1f2a95b758e76c5aec8138ed314d2ea823d3eff6afed0b8363a9760c5482e3b0"} Oct 31 02:13:39.082253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.082153 5756 scope.go:122] "RemoveContainer" containerID="312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9" Oct 31 02:13:39.092455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.092436 5756 scope.go:122] "RemoveContainer" containerID="312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9" Oct 31 02:13:39.092749 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:13:39.092729 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9\": container with ID starting with 312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9 not found: ID does not exist" containerID="312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9" Oct 31 02:13:39.092794 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.092760 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9"} err="failed to get container status \"312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9\": rpc error: code = NotFound desc = could not find container \"312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9\": container with ID starting with 312cd274d64c4c4a534456aad2d760ec5011cf91447502b2c9458a17bef01fa9 not found: ID does not exist" Oct 31 02:13:39.243826 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.243796 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp"] Oct 31 02:13:39.247223 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:39.247198 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b6f24b-6d9fd97cd5-zcbtp"] Oct 31 02:13:40.310639 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:40.310604 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" path="/var/lib/kubelet/pods/298b64a8-a86b-4988-bfb5-bd0467920df2/volumes" Oct 31 02:13:42.231386 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:42.231342 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerName="payload-processing" containerID="cri-o://f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce" gracePeriod=25 Oct 31 02:13:45.121307 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:45.121278 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m"] Oct 31 02:13:45.123749 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:45.123730 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-95404c-fdg5m"] Oct 31 02:13:46.310688 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:46.310654 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce6ea27-e7b9-44c1-8674-67d8f30f575a" path="/var/lib/kubelet/pods/fce6ea27-e7b9-44c1-8674-67d8f30f575a/volumes" Oct 31 02:13:47.205810 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:47.205769 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:48.259340 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:48.259303 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:13:51.715656 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:51.715623 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6"] Oct 31 02:13:51.716131 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:51.715930 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" containerID="cri-o://722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af" gracePeriod=30 Oct 31 02:13:56.978497 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:56.978473 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 02:13:57.144746 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.144715 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.144937 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.144769 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.144937 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.144899 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.144937 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.144930 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.145114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145004 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.145114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145039 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.145114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145040 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:57.145114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145079 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.145114 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145112 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.145389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145126 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:57.145389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145154 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls\") pod \"53f8af91-50d7-495a-a080-aa71bf3b68ed\" (UID: \"53f8af91-50d7-495a-a080-aa71bf3b68ed\") " Oct 31 02:13:57.145389 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145192 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:57.145564 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145404 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:57.145629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145578 5756 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f8af91-50d7-495a-a080-aa71bf3b68ed-istiod-ca-cert\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.145629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145597 5756 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-data\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.145629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145612 5756 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-credential-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.146626 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145628 5756 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-socket\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.146626 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.145925 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:57.147610 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.147580 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "kube-api-access-8jsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:57.147738 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.147721 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:57.148299 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.148278 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:57.148664 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.148641 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo" pod "53f8af91-50d7-495a-a080-aa71bf3b68ed" (UID: "53f8af91-50d7-495a-a080-aa71bf3b68ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:57.188825 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.188785 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v"] Oct 31 02:13:57.190648 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.190625 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t"] Oct 31 02:13:57.206073 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.206041 5756 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:57.206669 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.206645 5756 generic.go:356] "Generic (PLEG): container finished" podID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerID="722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af" exitCode=0 Oct 31 02:13:57.206784 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.206702 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" Oct 31 02:13:57.206784 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.206719 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" event={"ID":"53f8af91-50d7-495a-a080-aa71bf3b68ed","Type":"ContainerDied","Data":"722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af"} Oct 31 02:13:57.206784 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.206771 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6" event={"ID":"53f8af91-50d7-495a-a080-aa71bf3b68ed","Type":"ContainerDied","Data":"89ec910f3a7c26ec47e1a9218f4b5a6deab1709b6d9f178ef2bcaae2d955f96b"} Oct 31 02:13:57.206933 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.206801 5756 scope.go:122] "RemoveContainer" containerID="722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af" Oct 31 02:13:57.217642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.217626 5756 scope.go:122] "RemoveContainer" containerID="7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7" Oct 31 02:13:57.226062 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.226044 5756 scope.go:122] "RemoveContainer" containerID="722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af" Oct 31 02:13:57.226330 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:13:57.226312 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af\": container with ID starting with 722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af not found: ID does not exist" containerID="722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af" Oct 31 02:13:57.226379 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.226340 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af"} err="failed to get container status \"722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af\": rpc error: code = NotFound desc = could not find container \"722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af\": container with ID starting with 722712d68ad844f9c301323341a8cd5a900a5a7887d060c924126354a612f3af not found: ID does not exist" Oct 31 02:13:57.226379 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.226358 5756 scope.go:122] "RemoveContainer" containerID="7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7" Oct 31 02:13:57.226599 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:13:57.226581 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7\": container with ID starting with 7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7 not found: ID does not exist" containerID="7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7" Oct 31 02:13:57.226645 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.226605 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7"} err="failed to get container status \"7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7\": rpc error: code = NotFound desc = could not find container \"7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7\": container with ID starting with 7b51e9a0cadaf9c0b25f6bc2b05a99551703c03125126a85d2ceaf9d3c75e8f7 not found: ID does not exist" Oct 31 02:13:57.247074 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.247048 5756 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-envoy\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.247074 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.247070 5756 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-token\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.247208 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.247080 5756 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f8af91-50d7-495a-a080-aa71bf3b68ed-istio-podinfo\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.247208 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.247090 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jsls\" (UniqueName: \"kubernetes.io/projected/53f8af91-50d7-495a-a080-aa71bf3b68ed-kube-api-access-8jsls\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.247208 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.247103 5756 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f8af91-50d7-495a-a080-aa71bf3b68ed-workload-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:13:57.521536 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.521462 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6"] Oct 31 02:13:57.526462 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:57.526434 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-95404c-openshift-default-d65bc58dc-mw7n6"] Oct 31 02:13:58.310550 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:13:58.310516 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" path="/var/lib/kubelet/pods/53f8af91-50d7-495a-a080-aa71bf3b68ed/volumes" Oct 31 02:14:02.066822 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:14:02.066784 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:14:02.490436 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.490404 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 02:14:02.492778 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.492759 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl\" (UniqueName: \"kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl\") pod \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " Oct 31 02:14:02.492843 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.492818 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume\") pod \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\" (UID: \"17e49c4a-7ffd-44ad-9e6b-685f9ae30821\") " Oct 31 02:14:02.493168 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.493148 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume" pod "17e49c4a-7ffd-44ad-9e6b-685f9ae30821" (UID: "17e49c4a-7ffd-44ad-9e6b-685f9ae30821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:02.494967 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.494944 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl" pod "17e49c4a-7ffd-44ad-9e6b-685f9ae30821" (UID: "17e49c4a-7ffd-44ad-9e6b-685f9ae30821"). InnerVolumeSpecName "kube-api-access-7g4jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:02.593642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.593609 5756 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-plugins-config-volume\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:02.593642 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:02.593637 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4jl\" (UniqueName: \"kubernetes.io/projected/17e49c4a-7ffd-44ad-9e6b-685f9ae30821-kube-api-access-7g4jl\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:03.243687 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.243653 5756 generic.go:356] "Generic (PLEG): container finished" podID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerID="f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce" exitCode=0 Oct 31 02:14:03.244278 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.243700 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" Oct 31 02:14:03.244278 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.243733 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" event={"ID":"17e49c4a-7ffd-44ad-9e6b-685f9ae30821","Type":"ContainerDied","Data":"f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce"} Oct 31 02:14:03.244278 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.243803 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl" event={"ID":"17e49c4a-7ffd-44ad-9e6b-685f9ae30821","Type":"ContainerDied","Data":"dbc864a1882c355fb0d7e9beef9cc0cbef06e93ab6772f4e1ed10f44275fccd2"} Oct 31 02:14:03.244278 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.243836 5756 scope.go:122] "RemoveContainer" containerID="f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce" Oct 31 02:14:03.254612 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.254588 5756 scope.go:122] "RemoveContainer" containerID="f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce" Oct 31 02:14:03.254978 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:14:03.254938 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce\": container with ID starting with f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce not found: ID does not exist" containerID="f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce" Oct 31 02:14:03.255053 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.254986 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce"} err="failed to get container status \"f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce\": rpc error: code = NotFound desc = could not find container \"f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce\": container with ID starting with f62de94b78ad8183593846d8e7dca2c5b55a04de458431338e7cf9221aa030ce not found: ID does not exist" Oct 31 02:14:03.408334 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.408304 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl"] Oct 31 02:14:03.457800 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.457763 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-95404c-8589dcf5cc-58ktl"] Oct 31 02:14:03.928232 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928201 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x"] Oct 31 02:14:03.928783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928758 5756 cpu_manager.go:414] "containerMap: removing container" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerName="payload-processing" Oct 31 02:14:03.928783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928781 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerName="payload-processing" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928802 5756 cpu_manager.go:414] "containerMap: removing container" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928811 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928850 5756 cpu_manager.go:414] "containerMap: removing container" podUID="fce6ea27-e7b9-44c1-8674-67d8f30f575a" containerName="revoke-keys" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928876 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce6ea27-e7b9-44c1-8674-67d8f30f575a" containerName="revoke-keys" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928901 5756 cpu_manager.go:414] "containerMap: removing container" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928909 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928929 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928938 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928951 5756 cpu_manager.go:414] "containerMap: removing container" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerName="payload-processing" Oct 31 02:14:03.929065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.928959 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerName="payload-processing" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929093 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929109 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce6ea27-e7b9-44c1-8674-67d8f30f575a" containerName="revoke-keys" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929121 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f8af91-50d7-495a-a080-aa71bf3b68ed" containerName="istio-proxy" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929132 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929141 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="298b64a8-a86b-4988-bfb5-bd0467920df2" containerName="payload-processing" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929152 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" Oct 31 02:14:03.929569 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.929164 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" containerName="payload-processing" Oct 31 02:14:03.933350 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.933332 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:03.991098 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:03.991069 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x"] Oct 31 02:14:04.004904 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.004857 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.005060 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.004961 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.005060 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.005010 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhhm\" (UniqueName: \"kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.029563 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.029530 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 02:14:04.029699 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.029592 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-76qp4\"" Oct 31 02:14:04.106231 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.106198 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.106443 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.106252 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.106443 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.106287 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhhm\" (UniqueName: \"kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.106931 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.106906 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.108848 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.108830 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.163411 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.163386 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhhm\" (UniqueName: \"kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm\") pod \"maas-api-revoke-sub-2beba6ab-6577x\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.246590 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.246515 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:04.325418 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.325376 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e49c4a-7ffd-44ad-9e6b-685f9ae30821" path="/var/lib/kubelet/pods/17e49c4a-7ffd-44ad-9e6b-685f9ae30821/volumes" Oct 31 02:14:04.617473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:04.617407 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x"] Oct 31 02:14:04.620179 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:14:04.620151 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b3d585_8f43_4101_b5b0_2405e14ab746.slice/crio-5f5a59471b536e3f3ff6c0969ddd0ce98cc17ba7c8512a7626225964f84b02b3 WatchSource:0}: Error finding container 5f5a59471b536e3f3ff6c0969ddd0ce98cc17ba7c8512a7626225964f84b02b3: Status 404 returned error can't find the container with id 5f5a59471b536e3f3ff6c0969ddd0ce98cc17ba7c8512a7626225964f84b02b3 Oct 31 02:14:05.258630 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:05.258549 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" event={"ID":"51b3d585-8f43-4101-b5b0-2405e14ab746","Type":"ContainerStarted","Data":"df05adc042f87a59e560325a2344a003d4a7243b56e71440b8463876942c6caf"} Oct 31 02:14:05.258630 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:05.258602 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" event={"ID":"51b3d585-8f43-4101-b5b0-2405e14ab746","Type":"ContainerStarted","Data":"5f5a59471b536e3f3ff6c0969ddd0ce98cc17ba7c8512a7626225964f84b02b3"} Oct 31 02:14:05.372712 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:05.372654 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" podStartSLOduration=2.372641357 podStartE2EDuration="2.372641357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:14:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:14:05.371093424 +0000 UTC m=+4617.178060970" watchObservedRunningTime="2026-10-31 02:14:05.372641357 +0000 UTC m=+4617.179608899" Oct 31 02:14:06.265796 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:06.265757 5756 generic.go:356] "Generic (PLEG): container finished" podID="51b3d585-8f43-4101-b5b0-2405e14ab746" containerID="df05adc042f87a59e560325a2344a003d4a7243b56e71440b8463876942c6caf" exitCode=0 Oct 31 02:14:06.266236 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:06.265841 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" event={"ID":"51b3d585-8f43-4101-b5b0-2405e14ab746","Type":"ContainerDied","Data":"df05adc042f87a59e560325a2344a003d4a7243b56e71440b8463876942c6caf"} Oct 31 02:14:07.410161 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.410132 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:07.438844 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.438814 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca\") pod \"51b3d585-8f43-4101-b5b0-2405e14ab746\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " Oct 31 02:14:07.439023 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.438900 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm\" (UniqueName: \"kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm\") pod \"51b3d585-8f43-4101-b5b0-2405e14ab746\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " Oct 31 02:14:07.439090 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.439021 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca\") pod \"51b3d585-8f43-4101-b5b0-2405e14ab746\" (UID: \"51b3d585-8f43-4101-b5b0-2405e14ab746\") " Oct 31 02:14:07.439206 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.439184 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca" pod "51b3d585-8f43-4101-b5b0-2405e14ab746" (UID: "51b3d585-8f43-4101-b5b0-2405e14ab746"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:07.439299 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.439284 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-openshift-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:07.440882 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.440842 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm" pod "51b3d585-8f43-4101-b5b0-2405e14ab746" (UID: "51b3d585-8f43-4101-b5b0-2405e14ab746"). InnerVolumeSpecName "kube-api-access-xqhhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:07.441075 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.441053 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca" pod "51b3d585-8f43-4101-b5b0-2405e14ab746" (UID: "51b3d585-8f43-4101-b5b0-2405e14ab746"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:07.540783 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.540753 5756 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51b3d585-8f43-4101-b5b0-2405e14ab746-maas-api-xks-ca\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:07.540962 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:07.540789 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqhhm\" (UniqueName: \"kubernetes.io/projected/51b3d585-8f43-4101-b5b0-2405e14ab746-kube-api-access-xqhhm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:08.278779 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:08.278747 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" Oct 31 02:14:08.279029 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:08.278783 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2beba6ab-6577x" event={"ID":"51b3d585-8f43-4101-b5b0-2405e14ab746","Type":"ContainerDied","Data":"5f5a59471b536e3f3ff6c0969ddd0ce98cc17ba7c8512a7626225964f84b02b3"} Oct 31 02:14:08.279029 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:08.278828 5756 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f5a59471b536e3f3ff6c0969ddd0ce98cc17ba7c8512a7626225964f84b02b3" Oct 31 02:14:11.736652 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:11.736621 5756 scope.go:122] "RemoveContainer" containerID="c8d1e72d5e5756afb1d3391c0d84079861d0eef29f73e88fd9598c24d42cfc8d" Oct 31 02:14:11.746612 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:11.746579 5756 scope.go:122] "RemoveContainer" containerID="f74eb08716fdc8defcfc03105bb2763e8392f0cd0d0d2bd07ab69f2609f6e874" Oct 31 02:14:11.755244 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:11.755225 5756 scope.go:122] "RemoveContainer" containerID="dd43727f25daa06354c59075c0af0e002400f922b5e1a126bf2b5b0225e3915d" Oct 31 02:14:11.763402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:11.763388 5756 scope.go:122] "RemoveContainer" containerID="ed465996dc53f3050b923b987d0414577ef89c1f359a788f9a376c141570edf6" Oct 31 02:14:12.211010 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.210972 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" containerID="cri-o://9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e" gracePeriod=45 Oct 31 02:14:12.212895 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.212843 5756 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" containerID="cri-o://4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f" gracePeriod=45 Oct 31 02:14:12.471995 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.471968 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 02:14:12.474795 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.474776 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 02:14:12.486183 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486163 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs\") pod \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " Oct 31 02:14:12.486266 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486208 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm\") pod \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " Oct 31 02:14:12.486266 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486243 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir\") pod \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " Oct 31 02:14:12.486351 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486271 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz\" (UniqueName: \"kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz\") pod \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " Oct 31 02:14:12.486351 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486302 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home\") pod \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " Oct 31 02:14:12.486456 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486385 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache\") pod \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\" (UID: \"2a81b1d0-0c85-420f-ade0-2ace58b65dff\") " Oct 31 02:14:12.486568 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486525 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir" pod "2a81b1d0-0c85-420f-ade0-2ace58b65dff" (UID: "2a81b1d0-0c85-420f-ade0-2ace58b65dff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.486651 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486624 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home" pod "2a81b1d0-0c85-420f-ade0-2ace58b65dff" (UID: "2a81b1d0-0c85-420f-ade0-2ace58b65dff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.486691 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486659 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache" pod "2a81b1d0-0c85-420f-ade0-2ace58b65dff" (UID: "2a81b1d0-0c85-420f-ade0-2ace58b65dff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.486764 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486749 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.486805 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486767 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.486805 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.486776 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.488517 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.488494 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz" pod "2a81b1d0-0c85-420f-ade0-2ace58b65dff" (UID: "2a81b1d0-0c85-420f-ade0-2ace58b65dff"). InnerVolumeSpecName "kube-api-access-mkhcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:12.488971 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.488952 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm" pod "2a81b1d0-0c85-420f-ade0-2ace58b65dff" (UID: "2a81b1d0-0c85-420f-ade0-2ace58b65dff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.489040 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.488984 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs" pod "2a81b1d0-0c85-420f-ade0-2ace58b65dff" (UID: "2a81b1d0-0c85-420f-ade0-2ace58b65dff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:12.587490 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.587435 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home\") pod \"ebbdc099-30e9-48db-a755-05ebedb07f49\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.587749 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm\") pod \"ebbdc099-30e9-48db-a755-05ebedb07f49\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.587850 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache\") pod \"ebbdc099-30e9-48db-a755-05ebedb07f49\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.587917 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz\" (UniqueName: \"kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz\") pod \"ebbdc099-30e9-48db-a755-05ebedb07f49\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.587938 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home" pod "ebbdc099-30e9-48db-a755-05ebedb07f49" (UID: "ebbdc099-30e9-48db-a755-05ebedb07f49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.587982 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir\") pod \"ebbdc099-30e9-48db-a755-05ebedb07f49\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588005 5756 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs\") pod \"ebbdc099-30e9-48db-a755-05ebedb07f49\" (UID: \"ebbdc099-30e9-48db-a755-05ebedb07f49\") " Oct 31 02:14:12.588343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588136 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache" pod "ebbdc099-30e9-48db-a755-05ebedb07f49" (UID: "ebbdc099-30e9-48db-a755-05ebedb07f49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.588744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588376 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir" pod "ebbdc099-30e9-48db-a755-05ebedb07f49" (UID: "ebbdc099-30e9-48db-a755-05ebedb07f49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.588744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588419 5756 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-home\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.588744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588499 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a81b1d0-0c85-420f-ade0-2ace58b65dff-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.588744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588516 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a81b1d0-0c85-420f-ade0-2ace58b65dff-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.588744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588533 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhcz\" (UniqueName: \"kubernetes.io/projected/2a81b1d0-0c85-420f-ade0-2ace58b65dff-kube-api-access-mkhcz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.588744 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.588548 5756 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-model-cache\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.590305 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.590284 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs" pod "ebbdc099-30e9-48db-a755-05ebedb07f49" (UID: "ebbdc099-30e9-48db-a755-05ebedb07f49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:12.590384 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.590342 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm" pod "ebbdc099-30e9-48db-a755-05ebedb07f49" (UID: "ebbdc099-30e9-48db-a755-05ebedb07f49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:12.591509 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.591491 5756 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz" pod "ebbdc099-30e9-48db-a755-05ebedb07f49" (UID: "ebbdc099-30e9-48db-a755-05ebedb07f49"). InnerVolumeSpecName "kube-api-access-rljzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:12.689637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.689601 5756 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljzz\" (UniqueName: \"kubernetes.io/projected/ebbdc099-30e9-48db-a755-05ebedb07f49-kube-api-access-rljzz\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.689637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.689627 5756 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-tmp-dir\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.689637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.689636 5756 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebbdc099-30e9-48db-a755-05ebedb07f49-tls-certs\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:12.689637 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:12.689645 5756 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebbdc099-30e9-48db-a755-05ebedb07f49-dshm\") on node \"ip-10-0-136-249.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 02:14:13.308502 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.308469 5756 generic.go:356] "Generic (PLEG): container finished" podID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerID="9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e" exitCode=0 Oct 31 02:14:13.309008 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.308515 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" Oct 31 02:14:13.309008 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.308559 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" event={"ID":"ebbdc099-30e9-48db-a755-05ebedb07f49","Type":"ContainerDied","Data":"9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e"} Oct 31 02:14:13.309008 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.308606 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v" event={"ID":"ebbdc099-30e9-48db-a755-05ebedb07f49","Type":"ContainerDied","Data":"49852ad6ea90a9c10bd78c2179f056610007809c3761e29aba2afe96b30ccd23"} Oct 31 02:14:13.309008 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.308630 5756 scope.go:122] "RemoveContainer" containerID="9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e" Oct 31 02:14:13.309991 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.309973 5756 generic.go:356] "Generic (PLEG): container finished" podID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerID="4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f" exitCode=0 Oct 31 02:14:13.310082 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.310009 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" event={"ID":"2a81b1d0-0c85-420f-ade0-2ace58b65dff","Type":"ContainerDied","Data":"4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f"} Oct 31 02:14:13.310082 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.310026 5756 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" Oct 31 02:14:13.310189 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.310031 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t" event={"ID":"2a81b1d0-0c85-420f-ade0-2ace58b65dff","Type":"ContainerDied","Data":"93a85aee710425ecca067b2c8b687449179c54bc844e3506d0a99c5c907e2561"} Oct 31 02:14:13.319718 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.319685 5756 scope.go:122] "RemoveContainer" containerID="9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e" Oct 31 02:14:13.320119 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:14:13.320024 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e\": container with ID starting with 9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e not found: ID does not exist" containerID="9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e" Oct 31 02:14:13.320119 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.320052 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e"} err="failed to get container status \"9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e\": rpc error: code = NotFound desc = could not find container \"9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e\": container with ID starting with 9dc63eed3bf93b1e8331058c7a27328fa2307b6218497f783bff9e2ad226de0e not found: ID does not exist" Oct 31 02:14:13.320119 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.320069 5756 scope.go:122] "RemoveContainer" containerID="4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f" Oct 31 02:14:13.328620 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.328605 5756 scope.go:122] "RemoveContainer" containerID="4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f" Oct 31 02:14:13.328876 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:14:13.328845 5756 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f\": container with ID starting with 4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f not found: ID does not exist" containerID="4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f" Oct 31 02:14:13.328916 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.328883 5756 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f"} err="failed to get container status \"4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f\": rpc error: code = NotFound desc = could not find container \"4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f\": container with ID starting with 4e581a7ba2a0ea1cd1aa590cfbcdd50353a26682edf9003c0c79fe808d64071f not found: ID does not exist" Oct 31 02:14:13.469723 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.469683 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v"] Oct 31 02:14:13.518909 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.518819 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/distinct-w4-acd385-kserve-84bd5b7b5f-4ft4v"] Oct 31 02:14:13.627535 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.627503 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t"] Oct 31 02:14:13.628873 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:13.628839 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-acd385/simulator-w4-acd385-kserve-6698cbdc4f-xbg6t"] Oct 31 02:14:14.310776 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:14.310743 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" path="/var/lib/kubelet/pods/2a81b1d0-0c85-420f-ade0-2ace58b65dff/volumes" Oct 31 02:14:14.311194 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:14.311179 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" path="/var/lib/kubelet/pods/ebbdc099-30e9-48db-a755-05ebedb07f49/volumes" Oct 31 02:14:20.110096 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:20.110058 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:14:30.131470 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:30.131441 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qph5z_17eeee37-e7d5-470d-b743-39dd06a083c2/kuadrant-console-plugin/0.log" Oct 31 02:14:32.066040 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:14:32.066006 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:14:33.277754 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:33.277720 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wzhvt"] Oct 31 02:14:41.861402 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:41.861363 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wzhvt_7f35581f-76fb-4ba1-9671-38aed1743835/limitador/0.log" Oct 31 02:14:43.509355 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:43.509329 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:14:44.035473 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:44.035440 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:14:44.874917 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:44.874885 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:14:45.157441 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:45.157361 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:14:45.416646 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:45.416567 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:14:57.745592 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:57.745564 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:14:58.392487 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:14:58.392444 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:15:02.066272 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:15:02.066235 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:15:09.289022 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:09.288989 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:15:09.582117 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:09.582032 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:15:09.822639 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:09.822591 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:15:11.712006 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:11.711980 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1ba2ca5f-l7tn6_3035265c-b1fc-4ff6-b025-f02efc0f39a6/revoke-keys/0.log" Oct 31 02:15:11.816608 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:11.816577 5756 scope.go:122] "RemoveContainer" containerID="6997e2eefd7f89ae779fa18aa529069fda49d947e7cc8b5a9a5b33b3e7d68c06" Oct 31 02:15:11.825703 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:11.825679 5756 scope.go:122] "RemoveContainer" containerID="f253acaab7db8e0048cf75d58c79d8d9e29f30a4aa92e74e213398db4e4d173d" Oct 31 02:15:11.834190 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:11.834175 5756 scope.go:122] "RemoveContainer" containerID="6c1cbd2f06343b1d7b36c0bf6a2ff5c8df58dac4fda0d62937bd02f5f63e3411" Oct 31 02:15:12.191905 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:12.191881 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2beba6ab-6577x_51b3d585-8f43-4101-b5b0-2405e14ab746/revoke-keys/0.log" Oct 31 02:15:12.417526 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:12.417499 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3ffe7285-vtvwz_b047bf2e-8097-4295-995f-338a8fcacac6/revoke-keys/0.log" Oct 31 02:15:12.658150 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:12.658121 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6bcafe23-drvzz_c2f2ed87-350e-4306-a7cc-81c17a3dd2d0/revoke-keys/0.log" Oct 31 02:15:12.895939 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:12.895911 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6d2bd76d-2mjvm_3479d69c-2d01-4b7a-adfc-1c3ce2459a47/revoke-keys/0.log" Oct 31 02:15:13.134411 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:13.134383 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c1794932-4rvgk_fc396070-7cf4-4868-8a62-6ce18722318c/revoke-keys/0.log" Oct 31 02:15:13.380292 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:13.380258 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ced32f81-wjtfl_a9b13744-e69d-45d4-a8f4-c7ec207b29f8/revoke-keys/0.log" Oct 31 02:15:13.616392 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:13.616361 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-kxqm2_774af864-df87-4b5c-924d-4884df9e35da/postgres/0.log" Oct 31 02:15:16.298044 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:16.297973 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qph5z_17eeee37-e7d5-470d-b743-39dd06a083c2/kuadrant-console-plugin/0.log" Oct 31 02:15:17.783557 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:17.783528 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wzhvt_7f35581f-76fb-4ba1-9671-38aed1743835/limitador/0.log" Oct 31 02:15:19.726490 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:19.726441 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-52lbx_2a29063c-17ab-4027-80dd-ee3eced68a39/kube-auth-proxy/0.log" Oct 31 02:15:32.065730 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:15:32.065690 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:15:40.043248 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:40.043212 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm"] Oct 31 02:15:40.092417 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:40.092386 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d2bd76d-2mjvm"] Oct 31 02:15:40.310253 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:40.310176 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3479d69c-2d01-4b7a-adfc-1c3ce2459a47" path="/var/lib/kubelet/pods/3479d69c-2d01-4b7a-adfc-1c3ce2459a47/volumes" Oct 31 02:15:41.042720 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:41.042685 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk"] Oct 31 02:15:41.091994 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:41.091969 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1794932-4rvgk"] Oct 31 02:15:42.309144 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:42.309110 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc396070-7cf4-4868-8a62-6ce18722318c" path="/var/lib/kubelet/pods/fc396070-7cf4-4868-8a62-6ce18722318c/volumes" Oct 31 02:15:56.046611 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.046568 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl"] Oct 31 02:15:56.095797 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.095768 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced32f81-wjtfl"] Oct 31 02:15:56.097357 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.097327 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz"] Oct 31 02:15:56.098379 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.098364 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz"] Oct 31 02:15:56.099336 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.099316 5756 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6"] Oct 31 02:15:56.144974 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.144945 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ffe7285-vtvwz"] Oct 31 02:15:56.146107 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.146093 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bcafe23-drvzz"] Oct 31 02:15:56.147146 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.147128 5756 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ba2ca5f-l7tn6"] Oct 31 02:15:56.310215 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.310143 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3035265c-b1fc-4ff6-b025-f02efc0f39a6" path="/var/lib/kubelet/pods/3035265c-b1fc-4ff6-b025-f02efc0f39a6/volumes" Oct 31 02:15:56.310531 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.310518 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b13744-e69d-45d4-a8f4-c7ec207b29f8" path="/var/lib/kubelet/pods/a9b13744-e69d-45d4-a8f4-c7ec207b29f8/volumes" Oct 31 02:15:56.310884 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.310851 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b047bf2e-8097-4295-995f-338a8fcacac6" path="/var/lib/kubelet/pods/b047bf2e-8097-4295-995f-338a8fcacac6/volumes" Oct 31 02:15:56.311514 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:15:56.311500 5756 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f2ed87-350e-4306-a7cc-81c17a3dd2d0" path="/var/lib/kubelet/pods/c2f2ed87-350e-4306-a7cc-81c17a3dd2d0/volumes" Oct 31 02:16:02.065401 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:16:02.065363 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:16:11.868816 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:11.868779 5756 scope.go:122] "RemoveContainer" containerID="c680ce66ee7c53f10ef3b8a9e29c60baa71e2e883e5e4948bdb417320bf51640" Oct 31 02:16:11.877750 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:11.877727 5756 scope.go:122] "RemoveContainer" containerID="0c744f117ab345f6a4104642bdee926db656d702c80536d748df1192ad866e99" Oct 31 02:16:11.886027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:11.886006 5756 scope.go:122] "RemoveContainer" containerID="cac613e78449ca4fdecc9de246c3e3fb5cc513afaba93de7a44588d8d05ee561" Oct 31 02:16:13.702076 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:13.702042 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-48ddr_66bb82ee-c3c0-4719-8a69-04ac4ce085cd/global-pull-secret-syncer/0.log" Oct 31 02:16:17.309079 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:17.309046 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lttht_10f64c92-95d2-4fc4-add2-d174ec7fe24f/konnectivity-agent/0.log" Oct 31 02:16:18.825154 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:18.825117 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-249.us-west-1.compute.internal_e548c58b36505791ab3955482325fe49/haproxy/0.log" Oct 31 02:16:25.938538 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:25.938503 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:16:26.043038 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:26.043008 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:16:27.064326 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:27.064294 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:16:27.142813 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:27.142774 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:16:27.414153 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:27.414070 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:16:27.476434 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:27.476395 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:16:27.527041 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:27.527009 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:16:27.584662 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:27.584636 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:16:28.720455 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:28.720414 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:16:28.776107 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:28.776070 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:16:31.993944 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:31.993913 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:16:32.066652 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:16:32.066618 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:16:32.099785 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:32.099759 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:16:33.136279 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:33.136238 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:16:33.191038 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:33.191007 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:16:33.534598 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:33.534571 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:16:33.580443 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:33.580405 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:16:33.672548 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:33.672524 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:16:33.712309 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:33.712286 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:16:34.906722 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:34.906683 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:16:34.955238 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:34.955207 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:16:38.303587 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:38.303552 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:16:39.395968 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:39.395921 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:16:39.685500 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:39.685403 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:16:39.808333 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:39.808303 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:16:40.703038 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:40.702999 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:16:45.410523 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:45.410450 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-52lbx_2a29063c-17ab-4027-80dd-ee3eced68a39/kube-auth-proxy/0.log" Oct 31 02:16:49.656064 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:49.656027 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-52lbx_2a29063c-17ab-4027-80dd-ee3eced68a39/kube-auth-proxy/0.log" Oct 31 02:16:53.980700 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:53.980671 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qph5z_17eeee37-e7d5-470d-b743-39dd06a083c2/kuadrant-console-plugin/0.log" Oct 31 02:16:54.038507 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038476 5756 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g"] Oct 31 02:16:54.038902 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038887 5756 cpu_manager.go:414] "containerMap: removing container" podUID="51b3d585-8f43-4101-b5b0-2405e14ab746" containerName="revoke-keys" Oct 31 02:16:54.038977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038903 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b3d585-8f43-4101-b5b0-2405e14ab746" containerName="revoke-keys" Oct 31 02:16:54.038977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038918 5756 cpu_manager.go:414] "containerMap: removing container" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" Oct 31 02:16:54.038977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038924 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" Oct 31 02:16:54.038977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038947 5756 cpu_manager.go:414] "containerMap: removing container" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" Oct 31 02:16:54.038977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038953 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d7c3bf-fe2d-4829-ae99-0b5d3746d3a0" containerName="istio-proxy" Oct 31 02:16:54.038977 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038971 5756 cpu_manager.go:414] "containerMap: removing container" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" Oct 31 02:16:54.039176 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.038978 5756 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" Oct 31 02:16:54.039176 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.039045 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a81b1d0-0c85-420f-ade0-2ace58b65dff" containerName="main" Oct 31 02:16:54.039176 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.039054 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b3d585-8f43-4101-b5b0-2405e14ab746" containerName="revoke-keys" Oct 31 02:16:54.039176 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.039061 5756 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebbdc099-30e9-48db-a755-05ebedb07f49" containerName="main" Oct 31 02:16:54.041980 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.041963 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.086765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.086730 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-proc\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.086765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.086769 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhts6\" (UniqueName: \"kubernetes.io/projected/f36bb2e4-9952-4923-935b-69b419b922fe-kube-api-access-dhts6\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.086966 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.086882 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-podres\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.086966 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.086925 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-sys\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.087036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.087011 5756 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-lib-modules\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.093103 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.093081 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5dndx\"/\"openshift-service-ca.crt\"" Oct 31 02:16:54.093266 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.093247 5756 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5dndx\"/\"kube-root-ca.crt\"" Oct 31 02:16:54.142532 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.142500 5756 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5dndx\"/\"default-dockercfg-wd6kq\"" Oct 31 02:16:54.144005 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.143978 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g"] Oct 31 02:16:54.187953 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.187919 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-lib-modules\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188133 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188019 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-proc\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188133 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188051 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhts6\" (UniqueName: \"kubernetes.io/projected/f36bb2e4-9952-4923-935b-69b419b922fe-kube-api-access-dhts6\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188133 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188070 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-lib-modules\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188133 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188128 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-proc\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188359 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188137 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-podres\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188359 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188181 5756 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-sys\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188359 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188273 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-podres\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.188359 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.188301 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f36bb2e4-9952-4923-935b-69b419b922fe-sys\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.245281 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.245203 5756 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhts6\" (UniqueName: \"kubernetes.io/projected/f36bb2e4-9952-4923-935b-69b419b922fe-kube-api-access-dhts6\") pod \"perf-node-gather-daemonset-vhf2g\" (UID: \"f36bb2e4-9952-4923-935b-69b419b922fe\") " pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.354624 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.354587 5756 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:54.728200 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.728164 5756 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g"] Oct 31 02:16:54.732523 ip-10-0-136-249 kubenswrapper[5756]: W1031 02:16:54.732494 5756 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf36bb2e4_9952_4923_935b_69b419b922fe.slice/crio-bf0e7dbc73d2e5aec963f09270b18a6bf84bf8a47d95ec18abe5e41e03f09469 WatchSource:0}: Error finding container bf0e7dbc73d2e5aec963f09270b18a6bf84bf8a47d95ec18abe5e41e03f09469: Status 404 returned error can't find the container with id bf0e7dbc73d2e5aec963f09270b18a6bf84bf8a47d95ec18abe5e41e03f09469 Oct 31 02:16:54.900103 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:54.900073 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qph5z_17eeee37-e7d5-470d-b743-39dd06a083c2/kuadrant-console-plugin/0.log" Oct 31 02:16:55.176152 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:55.176117 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" event={"ID":"f36bb2e4-9952-4923-935b-69b419b922fe","Type":"ContainerStarted","Data":"0e38243a4faa7485d6257344a6dae2fbc892bf57095a810b1cdaa6d9b9eeabe5"} Oct 31 02:16:55.176647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:55.176177 5756 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" event={"ID":"f36bb2e4-9952-4923-935b-69b419b922fe","Type":"ContainerStarted","Data":"bf0e7dbc73d2e5aec963f09270b18a6bf84bf8a47d95ec18abe5e41e03f09469"} Oct 31 02:16:55.176647 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:55.176259 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:16:55.198117 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:55.198092 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wzhvt_7f35581f-76fb-4ba1-9671-38aed1743835/limitador/0.log" Oct 31 02:16:55.291597 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:55.291538 5756 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" podStartSLOduration=2.291519762 podStartE2EDuration="2.291519762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:55.291042967 +0000 UTC m=+4787.098010523" watchObservedRunningTime="2026-10-31 02:16:55.291519762 +0000 UTC m=+4787.098487306" Oct 31 02:16:56.385282 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:56.385252 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wzhvt_7f35581f-76fb-4ba1-9671-38aed1743835/limitador/0.log" Oct 31 02:16:59.375727 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:59.375700 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/alertmanager/0.log" Oct 31 02:16:59.788730 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:16:59.788698 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/config-reloader/0.log" Oct 31 02:17:00.202216 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:00.202128 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/kube-rbac-proxy-web/0.log" Oct 31 02:17:00.614004 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:00.613956 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/kube-rbac-proxy/0.log" Oct 31 02:17:01.026243 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:01.026154 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/kube-rbac-proxy-metric/0.log" Oct 31 02:17:01.191979 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:01.191947 5756 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5dndx/perf-node-gather-daemonset-vhf2g" Oct 31 02:17:01.439270 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:01.439232 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/prom-label-proxy/0.log" Oct 31 02:17:01.861069 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:01.861030 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9561ed9e-0e79-444d-936c-6177af70ab26/init-config-reloader/0.log" Oct 31 02:17:02.066032 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:17:02.065999 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:17:02.801095 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:02.801063 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lbrgs_7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/kube-state-metrics/0.log" Oct 31 02:17:03.253946 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:03.253854 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lbrgs_7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/kube-rbac-proxy-main/0.log" Oct 31 02:17:03.670264 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:03.670230 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-lbrgs_7da2d7d4-64fd-4ecf-a842-70e58f3a6bf2/kube-rbac-proxy-self/0.log" Oct 31 02:17:04.089045 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:04.089021 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cf56d4544-tzfbs_236b6fa3-674c-4790-951e-8b91cb51deee/metrics-server/0.log" Oct 31 02:17:04.505356 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:04.505281 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-x7tkk_73310d59-0bee-4124-ad47-b480d3544468/monitoring-plugin/0.log" Oct 31 02:17:07.463065 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:07.463038 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qw74k_a62692fc-3f77-478d-90d5-577b92632a39/node-exporter/0.log" Oct 31 02:17:07.875928 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:07.875904 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qw74k_a62692fc-3f77-478d-90d5-577b92632a39/kube-rbac-proxy/0.log" Oct 31 02:17:08.290106 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:08.290079 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qw74k_a62692fc-3f77-478d-90d5-577b92632a39/init-textfile/0.log" Oct 31 02:17:08.706152 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:08.706079 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dv9w7_fc981b83-c624-4524-95c0-950ec6a283ee/kube-rbac-proxy-main/0.log" Oct 31 02:17:09.119196 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:09.119148 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dv9w7_fc981b83-c624-4524-95c0-950ec6a283ee/kube-rbac-proxy-self/0.log" Oct 31 02:17:09.531094 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:09.531015 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-dv9w7_fc981b83-c624-4524-95c0-950ec6a283ee/openshift-state-metrics/0.log" Oct 31 02:17:09.954377 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:09.954347 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/prometheus/0.log" Oct 31 02:17:10.365369 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:10.365343 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/config-reloader/0.log" Oct 31 02:17:10.777085 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:10.777016 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/thanos-sidecar/0.log" Oct 31 02:17:11.189466 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.189443 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/kube-rbac-proxy-web/0.log" Oct 31 02:17:11.601297 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.601269 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/kube-rbac-proxy/0.log" Oct 31 02:17:11.915548 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.915466 5756 scope.go:122] "RemoveContainer" containerID="67517818d0287c8ac00609fca8adbd3ffa65062ae5b1f13488cd45e702918858" Oct 31 02:17:11.924533 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.924514 5756 scope.go:122] "RemoveContainer" containerID="71e6b56bc4aafdfda1c1fd5cb95867e2dba1f528e3740a4f5f4a8b0e87dd0bfb" Oct 31 02:17:11.933354 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.933335 5756 scope.go:122] "RemoveContainer" containerID="9ad3676f94110e06f70f63b80da761f0e5d3da8fb0b3a6406aa03e6aeb7d06a3" Oct 31 02:17:11.941358 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.941338 5756 scope.go:122] "RemoveContainer" containerID="d83ff3773d7cc8ee78788ea5d7e3d9c3ffd0406c88aca55dedd1bb4e54baaf9b" Oct 31 02:17:11.949733 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.949709 5756 scope.go:122] "RemoveContainer" containerID="4b812efff954d4a9708ac1ee5b87483d02588302ddececc92032444a71ae217d" Oct 31 02:17:11.957722 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.957701 5756 scope.go:122] "RemoveContainer" containerID="446dd0dc0fa2cc030202f23e873a53c9be317ceab5c1b072a6ad29fa5ea10971" Oct 31 02:17:11.965294 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.965274 5756 scope.go:122] "RemoveContainer" containerID="38db32335a2b0e6cdaf1ca7da00c75e42c9a04468aaa5527dbb09ccadccf2ecd" Oct 31 02:17:11.973466 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:11.973439 5756 scope.go:122] "RemoveContainer" containerID="2a1e2773f2dc7c794ed5757007655bbb66db4a02cec40c9e6fe9d658d0c45ecc" Oct 31 02:17:12.014409 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:12.014387 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/kube-rbac-proxy-thanos/0.log" Oct 31 02:17:12.428629 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:12.428603 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_634481e2-b617-4271-8d38-9793dc458bd0/init-config-reloader/0.log" Oct 31 02:17:12.848098 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:12.848070 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-l72fc_64d66bc1-aa19-496e-82c5-ec3bedaa9943/prometheus-operator/0.log" Oct 31 02:17:13.259422 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:13.259346 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-l72fc_64d66bc1-aa19-496e-82c5-ec3bedaa9943/kube-rbac-proxy/0.log" Oct 31 02:17:15.339343 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:15.339318 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69f8cc8d77-825m2_2c49c201-987e-4ea9-9bec-343a09101685/thanos-query/0.log" Oct 31 02:17:15.750765 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:15.750683 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69f8cc8d77-825m2_2c49c201-987e-4ea9-9bec-343a09101685/kube-rbac-proxy-web/0.log" Oct 31 02:17:16.164565 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:16.164534 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69f8cc8d77-825m2_2c49c201-987e-4ea9-9bec-343a09101685/kube-rbac-proxy/0.log" Oct 31 02:17:16.576115 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:16.576086 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69f8cc8d77-825m2_2c49c201-987e-4ea9-9bec-343a09101685/prom-label-proxy/1.log" Oct 31 02:17:16.988271 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:16.988198 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69f8cc8d77-825m2_2c49c201-987e-4ea9-9bec-343a09101685/kube-rbac-proxy-rules/0.log" Oct 31 02:17:17.402524 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:17.402498 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69f8cc8d77-825m2_2c49c201-987e-4ea9-9bec-343a09101685/kube-rbac-proxy-metrics/0.log" Oct 31 02:17:25.231465 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:25.231433 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5c96d49795-dlb8h_078342a7-6fc6-4308-aa92-2e259d681d57/console/0.log" Oct 31 02:17:32.065603 ip-10-0-136-249 kubenswrapper[5756]: E1031 02:17:32.065561 5756 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c7pm5" containerName="extractor" Oct 31 02:17:35.342027 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:35.341998 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wklrq_41b41813-6309-49be-bd2c-76b0820899af/dns/0.log" Oct 31 02:17:35.753553 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:35.753485 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wklrq_41b41813-6309-49be-bd2c-76b0820899af/kube-rbac-proxy/0.log" Oct 31 02:17:36.172763 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:36.172731 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-649pf_3537b0f9-a07f-423d-a9e2-8a3abc3c3916/dns-node-resolver/0.log" Oct 31 02:17:37.253639 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:37.253606 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-597b95cb84-tbnvv_53a1362f-5096-48b6-9d24-3957a917fb64/cert-manager-controller/0.log" Oct 31 02:17:37.620142 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:37.620116 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-wvgmm_50b677a9-83c9-490e-bcdb-558330c160c7/cert-manager-webhook/0.log" Oct 31 02:17:40.522285 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:40.522256 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7fcdb566-g2dj9_c5e5cdfe-93dd-438e-ae70-a4268581023b/manager/0.log" Oct 31 02:17:40.859164 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:40.859136 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7fcdb566-g2dj9_c5e5cdfe-93dd-438e-ae70-a4268581023b/copy-manifests/0.log" Oct 31 02:17:41.088468 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:41.088432 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrqjp_63419a67-aaca-408d-bcfa-7c302b749c9d/node-ca/0.log" Oct 31 02:17:43.231246 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:43.231220 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-skln4_269646f7-4e2b-41ba-9500-73d1f9874888/manager/0.log" Oct 31 02:17:44.322448 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:44.322414 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-bgs5p_b793557f-8c8e-4def-85b6-7aa0b2c5e9ee/manager/0.log" Oct 31 02:17:44.610127 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:44.610018 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6887cffc9f-lvrkf_be3278b6-5de5-44bd-b388-a869169d754b/manager/0.log" Oct 31 02:17:44.724613 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:44.724584 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-km9nx_b81442f8-6450-4b10-8689-f6116b037c66/server/0.log" Oct 31 02:17:45.626036 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:45.625985 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-2j77z_81369cfa-2441-4b29-8a48-bf452d96b522/manager/0.log" Oct 31 02:17:47.925315 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:47.925279 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-52lbx_2a29063c-17ab-4027-80dd-ee3eced68a39/kube-auth-proxy/0.log" Oct 31 02:17:48.243940 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:48.243831 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-52lbx_2a29063c-17ab-4027-80dd-ee3eced68a39/kube-auth-proxy/0.log" Oct 31 02:17:49.269317 ip-10-0-136-249 kubenswrapper[5756]: I1031 02:17:49.269287 5756 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d58c8f9d5-52lbx_2a29063c-17ab-4027-80dd-ee3eced68a39/kube-auth-proxy/0.log"